TrawlerFetchStatus

AI Overview😉

  • The potential purpose of this module is to provide detailed information about the outcome of a URL fetch attempt, including the reason for any errors that occurred. This helps Google's search algorithm to understand why a particular URL was not crawled or indexed correctly.
  • This module could impact search results by influencing how Google's algorithm handles URLs that return errors or are blocked by robots.txt. For example, if a URL returns a 404 error, the Reason field could provide more context about the error, which might affect how the algorithm decides to crawl or index the URL in the future. This could lead to more accurate or relevant search results for users.
  • To be more favorable for this function, a website could ensure that their URLs are correctly configured and return accurate HTTP status codes. This includes using proper 404 pages for non-existent content, implementing correct redirects, and ensuring that robots.txt files are correctly configured to allow or disallow crawling. By providing clear and accurate information about their URLs, websites can help Google's algorithm make better decisions about how to crawl and index their content.

Interesting Module? Vote 👇

Voting helps other researchers find interesting modules.

Current Votes: 0

GoogleApi.ContentWarehouse.V1.Model.TrawlerFetchStatus (google_api_content_warehouse v0.4.0)

Attributes

  • Reason (type: integer(), default: nil) - The Reason field gives further clarifying details about why or how the fetch had the given outcome. For instance, if State is URL_ERROR - was it a 404/NotFound or a DNS error? The Reason field is present iff State != URL_CRAWLED. For a given crawl status of URL_FOO, the Reason value will be one of the various FetchFooReason enum values from crawler/trawler/trawler_enums.proto
  • State (type: String.t, default: nil) - The State field describes the basic outcome of a fetch (URL_CRAWLED, URL_ROBOTED, URL_ERROR, etc). The value is one of the UrlStatusType enum values from crawler/trawler/trawler_enums.proto Note, there are several combinations of this Status/Reason tuple that could mean that your content is crawled or can be bucketed in a particular type of error. So instead of comparing the enumeration values manually, we suggest to use the predicate functions such as IsContentCrawled() provided in crawler/trawler/public/basictypes.h (see details there).

Summary

Types

t()

Functions

decode(value, options)

Unwrap a decoded JSON object into its complex fields.

Types

Link to this type

t()

@type t() :: %GoogleApi.ContentWarehouse.V1.Model.TrawlerFetchStatus{
  Reason: integer() | nil,
  State: String.t() | nil
}

Functions

Link to this function

decode(value, options)

@spec decode(struct(), keyword()) :: struct()

Unwrap a decoded JSON object into its complex fields.