IndexingConverterRobotsInfo

AI Overview😉

  • The potential purpose of this module is to collect and process robots.txt directives and meta tags from web pages, which provide instructions to search engine crawlers on how to handle the content. This module seems to be responsible for interpreting these directives and determining how the content should be indexed, archived, and displayed in search results.
  • This module could impact search results by influencing what content is included or excluded from the index, how it is displayed (e.g., with or without snippets or images), and how it is archived. It may also affect the ranking of pages that have specific robots.txt directives or meta tags. For example, a page with a "noindex" directive may be excluded from search results, while a page with a "nofollow" directive may not have its links crawled.
  • A website may change things to be more favorable for this function by ensuring that their robots.txt file and meta tags are correctly configured and aligned with their desired search engine behavior. This may include setting specific directives for crawling, indexing, and archiving, as well as using meta tags to control snippet and image preview lengths. Additionally, websites may want to ensure that their content is easily crawlable and indexable by search engines, and that they are not inadvertently blocking or restricting access to their content.

Interesting Module? Vote 👇

Voting helps other researchers find interesting modules.

Current Votes: 0

GoogleApi.ContentWarehouse.V1.Model.IndexingConverterRobotsInfo (google_api_content_warehouse v0.4.0)

Robots Info encapsulates all robots.txt or related information that we know about the Document collected from multiple sources such as HTTP headers, meta robots tags etc.

Attributes

  • contentExpiry (type: integer(), default: nil) - time in unix time format after which this content should not be shown in the results. This in inferred from the X-Robots-Tag HTTP header with unavailable_after: Do not use this field directly. There is a column called content_expiration in Alexandria that includes this and other signals.
  • convertToRobotedReason (type: String.t, default: nil) -
  • disallowedReason (type: integer(), default: nil) -
  • indexifembeddedReason (type: integer(), default: nil) - IMPORTANT: if you add a new field here, update the MergeRobotsInfo() function to merge the new field.
  • maxImagePreview (type: String.t, default: nil) - Max image preview restriction applied to this data. A value of THUMBNAIL_UNSPECIFIED can be treated as though there is no restriction.
  • maxSnippetLength (type: String.t, default: nil) - Max snippet preview restriction applied to this data. If this field is zero, it indicates that no snippet data can be displayed, therefore this field should be checked using has_max_snippet_length to determine if it was set.
  • noarchiveReason (type: integer(), default: nil) -
  • nofollowReason (type: integer(), default: nil) -
  • noimageframeoverlayReason (type: integer(), default: nil) -
  • noimageindexReason (type: integer(), default: nil) -
  • noindexReason (type: integer(), default: nil) - Bit map of RobotedReasons values. When set to a non-zero value, the document should not be indexed or archived etc. based on the name of the tag.
  • nopreviewReason (type: integer(), default: nil) -
  • nosnippetReason (type: integer(), default: nil) -
  • notranslateReason (type: integer(), default: nil) -

Summary

Types

t()

Functions

decode(value, options)

Unwrap a decoded JSON object into its complex fields.

Types

Link to this type

t()

@type t() :: %GoogleApi.ContentWarehouse.V1.Model.IndexingConverterRobotsInfo{
  contentExpiry: integer() | nil,
  convertToRobotedReason: String.t() | nil,
  disallowedReason: integer() | nil,
  indexifembeddedReason: integer() | nil,
  maxImagePreview: String.t() | nil,
  maxSnippetLength: String.t() | nil,
  noarchiveReason: integer() | nil,
  nofollowReason: integer() | nil,
  noimageframeoverlayReason: integer() | nil,
  noimageindexReason: integer() | nil,
  noindexReason: integer() | nil,
  nopreviewReason: integer() | nil,
  nosnippetReason: integer() | nil,
  notranslateReason: integer() | nil
}

Functions

Link to this function

decode(value, options)

@spec decode(struct(), keyword()) :: struct()

Unwrap a decoded JSON object into its complex fields.