TrawlerHostBucketData

AI Overview😉

  • The potential purpose of this module is to manage and track the load and capacity of web hosts, ensuring that Google's crawling and indexing processes do not overwhelm or abuse website resources. It appears to be a part of Google's effort to be a "good citizen" on the web and respect website owners' resources.
  • This module could impact search results by influencing how often and how deeply Google crawls a website. If a website is deemed to be overloaded or unable to handle additional traffic, Google may reduce the frequency or depth of its crawls, potentially affecting the website's visibility and ranking in search results. Conversely, websites with spare capacity may be crawled more frequently, potentially improving their search engine rankings.
  • A website may change things to be more favorable for this function by ensuring they have sufficient server resources and capacity to handle Google's crawls, implementing efficient caching and content delivery networks, and optimizing their website's architecture and infrastructure to handle increased traffic. Additionally, website owners may want to consider implementing crawl delay settings or robots.txt directives to communicate their preferences to Google's crawlers.

Interesting Module? Vote 👇

Voting helps other researchers find interesting modules.

Current Votes: 0

GoogleApi.ContentWarehouse.V1.Model.TrawlerHostBucketData (google_api_content_warehouse v0.4.0)

============================ Next Tag: 22 ============================ Data about the scheduling host bucket a URL was in (if the client wants to use this, e.g. for more intelligent scheduling, etc).

Attributes

  • ClientTrafficFraction (type: number(), default: nil) - How much existing traffic
  • ClientWeightFraction (type: number(), default: nil) - belong to the client How much weight the client
  • CurrentActiveConnections (type: integer(), default: nil) - How many connections are actively used for downloading ?
  • IsFull (type: boolean(), default: nil) - Is this bucket currently full ?
  • LastScheduleIntervalMs (type: String.t, default: nil) - How many ms ago we last scheduled a url
  • MaxActiveConnections (type: number(), default: nil) - The current hostload value (# of connections) - if negative does not apply
  • MediumTermLoad (type: number(), default: nil) - The load the recent times (the actual hostload that we apply is MaxActiveConnections / MediumTermLoad) - a hostload of 1.00 is normal, while over 1.0 is higher than normal load
  • MinInterRequestSecs (type: number(), default: nil) - The min delay between requests (in secs) - if negative does not apply
  • NonFullIntervalMs (type: String.t, default: nil) - If is full, when is becoming non-full (in ms)
  • TotalCapacityQps (type: number(), default: nil) - The following four fields attempt to make things simpler for clients to estimate available capacity. They are not populated yet as of 2013/08/21. Even after they are populated, they may change. So talk to trawler-dev@ before you use the fields. Total qps for this hostid
  • TotalUsedQps (type: number(), default: nil) - Currently used qps
  • urllist (type: list(GoogleApi.ContentWarehouse.V1.Model.TrawlerHostBucketDataUrlList.t), default: nil) -

Summary

Types

t()

Functions

decode(value, options)

Unwrap a decoded JSON object into its complex fields.

Types

Link to this type

t()

@type t() :: %GoogleApi.ContentWarehouse.V1.Model.TrawlerHostBucketData{
  ClientTrafficFraction: number() | nil,
  ClientWeightFraction: number() | nil,
  CurrentActiveConnections: integer() | nil,
  IsFull: boolean() | nil,
  LastScheduleIntervalMs: String.t() | nil,
  MaxActiveConnections: number() | nil,
  MediumTermLoad: number() | nil,
  MinInterRequestSecs: number() | nil,
  NonFullIntervalMs: String.t() | nil,
  TotalCapacityQps: number() | nil,
  TotalUsedQps: number() | nil,
  urllist:
    [GoogleApi.ContentWarehouse.V1.Model.TrawlerHostBucketDataUrlList.t()] | nil
}

Functions

Link to this function

decode(value, options)

@spec decode(struct(), keyword()) :: struct()

Unwrap a decoded JSON object into its complex fields.