Voting helps other researchers find interesting modules.
Trawler can add a policy label to a FetchReply. The two main cases are: - "spam" label added for specific spammer IPs listed in trawler_site_info, which most crawls auto-reject. - "roboted:useragent" (e.g. "roboted:googlebot") if InfoOnlyUserAgents field is set in FetchParams
ExtraData
integer()
nil
Label
String.t
Unwrap a decoded JSON object into its complex fields.
@type t() :: %GoogleApi.ContentWarehouse.V1.Model.TrawlerPolicyData{ ExtraData: integer() | nil, Label: String.t() | nil }
@spec decode(struct(), keyword()) :: struct()