Internet-Draft                                                F. Batum
Intended status: Standards Track                         April 6, 2026
Expires: October 8, 2026


        AI Discovery and Retrieval Endpoint (AIDRE)
                   draft-batum-aidre-00


Abstract

   This document specifies the AI Discovery and Retrieval Endpoint
   (AIDRE), a protocol for publishing machine-oriented, canonical, and
   semantically retrievable content on the web. AIDRE defines a
   discovery document, collection metadata, retrieval interfaces,
   optional vector-native query support, and content representation
   rules for AI systems.

   AIDRE aims to reduce redundant crawling, parsing, tokenization, and
   embedding of the same origin content while improving freshness,
   provenance, and interoperability for AI systems.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF). Note that other groups may also distribute working
   documents as Internet-Drafts. The list of current Internet-Drafts is
   at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on October 8, 2026.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.

Table of Contents

   1.  Introduction
   2.  Conventions and Terminology
   3.  Discovery

   4.  Collections
   5.  Query Model
   6.  Search Endpoint
   7.  Results
   8.  Chunk Retrieval
   9.  Caching and Freshness
   10. Error Handling
   11. Security Considerations
   12. Privacy Considerations
   13. IANA Considerations
   14. Media Types
   15. HTTP Usage
   16. Request and Response Envelope
   17. Error Model
   18. Pagination and Limits
   19. Embedding Compatibility Rules
   20. Representation Negotiation
   21. Rate Limiting
   22. Schema and OpenAPI Conformance
   23. Versioning and Extensibility
   24. HTTP Exchange Examples
   25. ABNF Summary
   26. References
   27. Additional Design Considerations
   Author's Address

1.  Introduction


   Modern AI retrieval systems repeatedly process the same web content
   by crawling, extracting, chunking, and embedding text. This results
   in redundant computation, increased costs, and inconsistent
   interpretations.

   AIDRE introduces a standardized mechanism for origins to expose
   canonical, machine-oriented retrieval interfaces, enabling direct
   semantic access to content.

2.  Conventions and Terminology


   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.1.  Terms


   Origin: The web origin publishing AIDRE metadata.

   Discovery Document: A JSON document describing endpoints and
   capabilities.

   Collection: A logical grouping of content.


   Chunk: A retrievable unit of canonical content.

   Embedding Space: A defined vector representation space.

3.  Discovery

3.1.  Well-Known URI


   An origin implementing AIDRE MUST expose the discovery document at

      /.well-known/ai-discovery


   over HTTPS using a well-known URI as defined in [RFC8615].

3.2.  Media Type


   The discovery document MUST be served as application/json.

3.3.  DNS Considerations


   Clients MUST NOT require DNS SRV lookup for discovery.

   DNS HTTPS (SVCB) records MAY be used as optimization hints, as
   described in [RFC9460].

   DNS SRV records are NOT RECOMMENDED for public interoperability.

3.4.  Example


   {
     "version": "1",
     "service": "AIDRE",
     "endpoints": {
       "search": "https://ai.example.com/search",
       "collections": "https://ai.example.com/collections",
       "chunk": "https://ai.example.com/chunks/{id}"
     },
     "capabilities": {
       "query_text": true,
       "query_vector": true
     }
   }

4.  Collections


   Collections group retrievable content.

   Each collection SHOULD define:


   *  name
   *  description
   *  visibility
   *  updated_at

5.  Query Model

5.1.  General Rule


   A request MUST include exactly one of:

   *  query
   *  query_vector

5.2.  Text Query


   Servers MUST support text queries.

5.3.  Vector Query


   Servers SHOULD support vector queries.

   If used, clients MUST specify embedding_space.

6.  Search Endpoint

6.1.  Method


   Clients MUST use HTTP POST.

6.2.  Request Example


   {
     "query": "sso setup",
     "collection": "docs",
     "top_k": 5
   }

6.3.  Vector Example


   {
     "query_vector": [0.01, -0.02],
     "embedding_space": "example-space",
     "collection": "docs"
   }

7.  Results

7.1.  Structure



   Responses MUST include a results array.

7.2.  Example


   {
     "results": [
       {
         "id": "doc_1#chunk_1",
         "score": 0.91,
         "metadata": {
           "updated_at": "2026-04-01T00:00:00Z",
           "canonical": true
         }
       }
     ]
   }

7.3.  Optional Fields


   Servers MAY return:

   *  text
   *  semantic payload
   *  vectors

   Servers SHOULD NOT return vectors by default.

8.  Chunk Retrieval


   Servers SHOULD support:

      GET /chunks/{id}


9.  Caching and Freshness


   Servers SHOULD expose:

   *  content_hash
   *  updated_at
   *  etag

10.  Error Handling


   Errors MUST be JSON objects.

   Example:


   {

     "error": "unsupported_embedding_space",
     "message": "Embedding space not supported"
   }

11.  Security Considerations


   AIDRE can make machine-oriented retrieval of content more efficient.
   That efficiency introduces risks, including accelerated scraping,
   exfiltration of structured content, abuse of vector-return paths,
   stale or malicious publication, and misleading provenance.

11.1.  Retrieval Amplification


   AIDRE can reduce the cost of large-scale content extraction for both
   legitimate and illegitimate clients. Deployments SHOULD consider rate
   limiting, authentication, authorization, abuse detection, and staged
   disclosure of higher-value representations.

11.2.  Vector Disclosure and Leakage


   Returning vectors can create additional disclosure risk beyond
   returning text alone. Depending on the embedding model, vectors MAY
   expose distributional or structural properties of source content and
   MAY enable downstream correlation, approximate membership inference,
   or other forms of analysis not intended by the publisher.

   Accordingly:

   *  Servers SHOULD NOT return vectors by default.
   *  Servers SHOULD evaluate whether vector return is necessary for a
      given deployment.
   *  Sensitive or access-controlled collections SHOULD disable vector
      return unless there is a specific operational need.
   *  Deployments MAY use distinct policies for public and authenticated
      collections.

   This specification does not require or assume that vector inversion
   is practical in all settings, but deployments SHOULD treat vector
   disclosure as a potentially sensitive operation.

11.3.  Malicious or Stale Canonical Content


   AIDRE gives publishers a machine-oriented canonical surface. If that
   surface is compromised, stale, poisoned, or maliciously altered,
   downstream systems MAY ingest or trust incorrect content at scale.

   Deployments SHOULD therefore consider content review workflows,
   publication approvals, deprecation semantics, rollback procedures,
   and freshness validation.

11.4.  Signed Provenance



   Deployments MAY attach signed provenance metadata to responses,
   collections, or chunk resources. Signed provenance can help clients
   verify origin authenticity, content integrity, and publication scope.

   If signed provenance is used, deployments SHOULD define:

   *  what is signed,
   *  which keys are authoritative,
   *  how keys are discovered,
   *  signature lifetime and rotation policy,
   *  the failure behavior when signature validation does not succeed.

   A deployment using signed provenance SHOULD ensure that key discovery
   is bound to the publisher's trust model, for example via HTTPS on the
   publisher origin or another integrity-protected mechanism.

11.5.  Query Abuse


   Query inputs, including query vectors, can be adversarial. Servers
   SHOULD validate input size, dimensionality, and request shape, and
   SHOULD protect retrieval infrastructure against resource exhaustion.

11.6.  Access-Controlled Collections


   AIDRE does not imply that all collections are public. Private or
   tenant-scoped collections MUST be protected by appropriate
   authorization controls. Discovery documents SHOULD avoid revealing
   unnecessary detail about protected collections.

12.  Privacy Considerations


   Sensitive data SHOULD NOT be exposed unintentionally.

13.  IANA Considerations


   This document does not itself register a new media type.

   However, this specification defines and uses the label
   application/aidre+json as the preferred representation for AIDRE
   request and response bodies. If AIDRE proceeds toward broader
   standardization and deployment, a future revision of this document or
   a companion specification SHOULD request IANA registration for that
   media type in accordance with the procedures applicable at that time.

   Until such registration occurs, implementations SHOULD treat
   application/aidre+json as a provisional media type for experimental
   and pre-standard deployment.

14.  Media Types

14.1.  AIDRE JSON Media Type



   AIDRE request and response bodies SHOULD use the media type
   application/aidre+json.

   Clients MUST send a Content-Type header of application/aidre+json
   when sending AIDRE request bodies.

   Clients SHOULD send an Accept header of application/aidre+json.

   Servers MAY accept application/json for compatibility, but a
   conformant implementation SHOULD prefer application/aidre+json.

14.2.  Character Encoding


   AIDRE JSON payloads MUST use UTF-8 encoding.

15.  HTTP Usage

15.1.  Methods


   The discovery and collection resources MUST support HTTP GET.

   The search resource MUST support HTTP POST.

   The chunk dereference resource MUST support HTTP GET.

15.2.  Request Headers


   Clients SHOULD send:

   *  Accept: application/aidre+json

   Clients sending request bodies MUST send:

   *  Content-Type: application/aidre+json

15.3.  Response Headers


   Servers SHOULD include a Content-Type response header set to
   application/aidre+json for successful AIDRE responses.

   Servers MAY include ETag, Last-Modified, Cache-Control, and RateLimit
   fields where appropriate.

15.4.  Status Codes


   Servers SHOULD use the following HTTP status codes consistently:

   *  200 OK: successful request.
   *  201 Created: resource created, if an extension defines creation.

   *  400 Bad Request: syntactically invalid request.
   *  401 Unauthorized: authentication required or failed.
   *  403 Forbidden: authenticated but not permitted.
   *  404 Not Found: resource identifier not found.
   *  409 Conflict: request conflicts with resource state.
   *  415 Unsupported Media Type: unsupported Content-Type.
   *  422 Unprocessable Entity: semantically invalid request, such as
      unsupported embedding dimensionality.
   *  429 Too Many Requests: rate limit exceeded.
   *  500 Internal Server Error: unexpected server error.
   *  503 Service Unavailable: temporary overload or maintenance.

16.  Request and Response Envelope

16.1.  General Response Shape


   A successful search response MUST be a JSON object containing a
   results member whose value is an array.

   A successful response SHOULD also contain:

   *  request_id
   *  collection
   *  meta

16.2.  Response Envelope Example


   {
     "request_id": "req_7f4c1c",
     "collection": "docs",
     "results": [
       {
         "id": "doc_123#chunk_7",
         "score": 0.92,
         "metadata": {
           "updated_at": "2026-04-01T10:00:00Z",
           "canonical": true,
           "visibility": "public"
         }
       }
     ],
     "meta": {
       "returned": 1,
       "top_k": 3
     }
   }

16.3.  Request Identifiers


   Servers SHOULD generate a request identifier for each request.

   A request identifier SHOULD be unique within operationally relevant
   scope and SHOULD be suitable for debugging and audit correlation.


17.  Error Model

17.1.  Error Envelope


   Errors MUST be JSON objects.

   Error responses SHOULD contain:

   *  error
   *  message
   *  details
   *  request_id

17.2.  Error Example


   {
     "error": "unsupported_embedding_space",
     "message": "The requested embedding space is not supported.",
     "details": {
       "embedding_space": "example:unsupported-space"
     },
     "request_id": "req_7f4c1c"
   }

17.3.  Error Codes


   Servers SHOULD use stable, machine-readable error values. Suggested
   values include:

   *  invalid_request
   *  unsupported_embedding_space
   *  invalid_embedding_dimension
   *  unsupported_return_field
   *  unauthorized
   *  forbidden
   *  not_found
   *  rate_limited
   *  internal_error

18.  Pagination and Limits

18.1.  top_k Semantics


   The top_k member requests the maximum number of results returned.

   If top_k is omitted, the server MUST apply a documented default.

   Servers MUST reject non-positive top_k values.

   Servers MAY enforce an implementation-defined maximum top_k.


   If the client requests a top_k larger than the server maximum, the
   server MUST either reject the request with an error or clamp the
   value according to documented behavior.

18.2.  Pagination


   Search endpoints MAY support pagination.

   If pagination is supported, the server SHOULD use an opaque cursor
   scheme.

   The client MUST treat cursor values as opaque.

18.3.  Pagination Envelope Example


   {
     "request_id": "req_9aa1",
     "collection": "docs",
     "results": [],
     "meta": {
       "returned": 0,
       "top_k": 10,
       "next_cursor": "eyJvZmZzZXQiOjEwMH0"
     }
   }

19.  Embedding Compatibility Rules

19.1.  Embedding Space Identification


   Each embedding space advertised by the server MUST include:

   *  id
   *  dimensions
   *  distance

   It SHOULD also include:

   *  normalized
   *  provider
   *  model
   *  revision

19.2.  Dimensionality


   If a query_vector length does not match the dimensions declared for
   the requested embedding space, the server MUST reject the request
   with 422 Unprocessable Entity.

19.3.  Distance Function



   The server MUST evaluate query vectors according to the distance
   function declared by the embedding space.

   Clients MUST NOT assume cosine similarity unless the embedding space
   declaration says so.

19.4.  Normalization


   If an embedding space declares normalized=true, clients SHOULD send
   normalized vectors.

   A server MAY normalize vectors on receipt, but such behavior SHOULD
   be documented.

19.5.  Compatibility Failure Example


   {
     "error": "invalid_embedding_dimension",
     "message": "The supplied query vector has length 1024, but the
       embedding space requires 3072.",
     "details": {
       "embedding_space":
         "openai:text-embedding-3-large:3072:cosine:v1",
       "expected_dimensions": 3072,
       "actual_dimensions": 1024
     }
   }

20.  Representation Negotiation

20.1.  Return Object


   The return object requests optional fields in result members.

   Supported fields MAY include:

   *  ids
   *  metadata
   *  text
   *  semantic_payload
   *  vectors

20.2.  Server Behavior


   If a client requests an unsupported return field, the server MUST
   either:

   *  ignore the field, or
   *  reject the request with a machine-readable error.

   The chosen behavior MUST be documented by the deployment.


20.3.  Unknown Fields


   Clients MUST ignore unknown response fields.

   Servers SHOULD ignore unknown request fields unless doing so would
   create ambiguous or unsafe behavior.

21.  Rate Limiting

21.1.  General


   Servers MAY apply rate limiting.

   If rate limiting is applied, servers SHOULD expose limit state using
   HTTP rate limit fields defined in [RFC9333].

21.2.  Suggested Fields


   Deployments SHOULD consider exposing:

   *  RateLimit-Limit
   *  RateLimit-Remaining
   *  RateLimit-Reset

21.3.  Related Specifications


   Deployments are encouraged to align with existing HTTP rate limit
   field specifications rather than inventing deployment-specific
   headers when interoperable fields are sufficient.

22.  Schema and OpenAPI Conformance

22.1.  Companion Artifacts


   This specification MAY be accompanied by JSON Schema, OpenAPI, or
   similar machine-readable descriptions of AIDRE messages and
   resources.

22.2.  Normative Authority


   Such artifacts are useful for tooling, validation, code generation,
   testing, and documentation. However, unless explicitly stated
   otherwise, this document remains the normative definition of the
   protocol.

   If an OpenAPI description, JSON Schema, or other companion artifact
   conflicts with this document, this document takes precedence.

22.3.  Conformance Guidance



   Deployments SHOULD ensure that published companion artifacts are kept
   consistent with the protocol version advertised in the discovery
   document.

23.  Versioning and Extensibility

23.1.  Discovery Version


   The discovery document MUST contain a version member.

23.2.  Compatibility Rule


   Backward-compatible additions SHOULD be made by adding new fields.

   Clients MUST ignore unknown fields.

   Backward-incompatible changes SHOULD be introduced through a new
   protocol version.

23.3.  Extension Members


   Deployments MAY define extension members.

   Extension members SHOULD use names that minimize collision risk.

24.  HTTP Exchange Examples

24.1.  Discovery Request Example

   Request:


   GET /.well-known/ai-discovery HTTP/1.1
   Host: example.com
   Accept: application/aidre+json, application/json

   Response:

   HTTP/1.1 200 OK
   Content-Type: application/aidre+json
   Cache-Control: max-age=300
   ETag: "disc-v1-9f2a"

   {
     "version": "1",
     "service": "AIDRE",
     "organization": "Example Corp",
     "endpoints": {
       "collections": "https://ai.example.com/collections",
       "search": "https://ai.example.com/search",
       "chunk": "https://ai.example.com/chunks/{id}"
     },

     "capabilities": {
       "query_text": true,
       "query_vector": true,
       "return_text": true,
       "return_semantic_payload": true,
       "return_vectors": false,
       "delta_sync": false
     },
     "embedding_spaces": [
       {
         "id": "openai:text-embedding-3-large:3072:cosine:v1",
         "dimensions": 3072,
         "distance": "cosine",
         "normalized": true
       }
     ],
     "auth": {
       "type": "none"
     }
   }

24.2.  Search Request Example

   Request:


   POST /search HTTP/1.1
   Host: ai.example.com
   Accept: application/aidre+json
   Content-Type: application/aidre+json

   {
     "query_vector": [0.013, -0.028, 0.442],
     "embedding_space":
       "openai:text-embedding-3-large:3072:cosine:v1",
     "collection": "docs",
     "top_k": 3,
     "return": {
       "ids": true,
       "metadata": true,
       "text": false,
       "semantic_payload": true,
       "vectors": false
     }
   }

   Response:

   HTTP/1.1 200 OK
   Content-Type: application/aidre+json
   RateLimit-Limit: 100
   RateLimit-Remaining: 99
   RateLimit-Reset: 60

   {
     "request_id": "req_7f4c1c",

     "collection": "docs",
     "results": [
       {
         "id": "doc_123#chunk_7",
         "score": 0.92,
         "source": {
           "url": "https://example.com/docs/sso/setup",
           "title": "SSO Setup",
           "section": "Prerequisites"
         },
         "metadata": {
           "updated_at": "2026-04-01T10:00:00Z",
           "canonical": true,
           "visibility": "public",
           "content_hash": "sha256:abcd..."
         },
         "semantic_payload": {
           "type": "application/semantic+json",
           "claims": [
             {
               "subject": "SAML SSO",
               "predicate": "requires",
               "object": "domain verification"
             }
           ]
         }
       }
     ],
     "meta": {
       "returned": 1,
       "top_k": 3
     }
   }

24.3.  Compatibility Error Example

   Request:


   POST /search HTTP/1.1
   Host: ai.example.com
   Accept: application/aidre+json
   Content-Type: application/aidre+json

   {
     "query_vector": [0.1, 0.2],
     "embedding_space":
       "openai:text-embedding-3-large:3072:cosine:v1",
     "collection": "docs"
   }

   Response:

   HTTP/1.1 422 Unprocessable Entity
   Content-Type: application/aidre+json


   {
     "error": "invalid_embedding_dimension",
     "message": "The supplied query vector has length 2, but the
       embedding space requires 3072.",
     "details": {
       "embedding_space":
         "openai:text-embedding-3-large:3072:cosine:v1",
       "expected_dimensions": 3072,
       "actual_dimensions": 2
     },
     "request_id": "req_err_11"
   }

25.  ABNF Summary


   The following ABNF, using the notation from [RFC5234], summarizes the
   principal AIDRE JSON member names. It is descriptive of member names
   and presence expectations and is not a complete JSON grammar.

   query-member           = %s"query"
   query-vector-member    = %s"query_vector"
   embedding-space-member = %s"embedding_space"
   collection-member      = %s"collection"
   topk-member            = %s"top_k"
   return-member          = %s"return"
   results-member         = %s"results"
   metadata-member        = %s"metadata"
   request-id-member      = %s"request_id"
   next-cursor-member     = %s"next_cursor"

26.  References

26.1.  Normative References


   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", RFC 8174.

   [RFC5234]  Crocker, D., and P. Overell, "Augmented BNF for Syntax
              Specifications", RFC 5234.

   [RFC8615]  Nottingham, M., "Well-Known Uniform Resource
              Identifiers (URIs)", RFC 8615.

26.2.  Informative References

   [RFC9460]  Schwartz, B., "Service Binding and Parameter
              Specification via the DNS", RFC 9460.

   [RFC9333]  Polli, R. and M. Martinez, "The RateLimit Fields for
              HTTP", RFC 9333.


27.  Additional Design Considerations

27.1.  Why Text Query Remains Mandatory


   Text query support remains mandatory for baseline interoperability.
   Not all clients and deployments will share an embedding space, and a
   text path allows a client to interact with an AIDRE deployment even
   when vector compatibility has not been negotiated.

27.2.  Why Vector Return Is Optional


   Vector-native querying and vector disclosure are distinct concerns. A
   deployment may wish to support vector-native query semantics while
   refusing to disclose stored vectors. This separation allows the
   protocol to reduce redundant text processing without forcing a single
   disclosure model.

Author's Address


   Fatih Batum
   Istanbul, Turkiye
   Email: fatih@batum.gen.tr

