Concordancer

A concordancer is a computer program that automatically constructs a concordance—an alphabetised index of every occurrence of a word or phrase in a body of text, each entry displayed with its surrounding context. Concordancers are primary tools in corpus linguistics, lexicography, computer-assisted translation, and language teaching. The most common display format is the key word in context (KWIC) layout, in which each hit appears centred on a line with a fixed span of words to its left and right, enabling rapid scanning of usage patterns across many occurrences. == History == === Pre-computational concordances === The compilation of concordances predates computers by many centuries. Around 1230, the French Dominican cardinal Hugh of Saint-Cher directed a team of friars in assembling a concordance of the Latin Vulgate Bible, generally regarded as the first systematic concordance of any text. To help readers locate passages, Hugh divided each biblical chapter into lettered sections. Later milestones include a Hebrew Old Testament concordance compiled by Rabbi Mordecai Nathan (1448), Alexander Cruden's Complete Concordance to the Holy Scriptures (1737), and the manuscript Asaf ha-Mazkir, an unfinished concordance to the Babylonian Talmud compiled by Moses Rigotz around the turn of the 19th century. === First computer concordance === The first concordance produced with computing assistance was the Index Thomisticus, a comprehensive lexical index of the writings of and around Thomas Aquinas, totalling approximately 10.6 million Latin words. The Italian Jesuit priest Roberto Busa conceived the project in 1946 and secured the sponsorship of IBM in 1949 after a meeting with chairman Thomas J. Watson. Keypunch operators in Gallarate, Italy, encoded the texts onto punched cards from around 1950. IBM executive Paul Tasman developed the processing methods. The full 56-volume printed edition was completed around 1980, followed by a CD-ROM edition in 1989 and a web-accessible version in 2005. === The KWIC format === The key word in context (KWIC) display was formalised as a computational technique by Hans Peter Luhn, a researcher at IBM, in a 1960 paper in American Documentation. In KWIC output, each instance of the search term (the node word) is centred on a line with a fixed window of words to each side; sorting the resulting lines alphabetically by the immediately adjacent word reveals collocational and phraseological patterns at a glance. === COCOA === One of the first dedicated concordancing programs was COCOA (COunt and COncordance Generation on Atlas), created in 1965 by D. B. Russell at University College London and the Atlas Computer Laboratory in Harwell, Oxfordshire. Written in approximately 4,000 cards of FORTRAN, it processed text annotated with flat, non-hierarchical markup tags and could produce word counts and concordances in multiple languages. Within its first six months COCOA had been applied to texts in at least six languages. A second version designed for multiple mainframe platforms was distributed to British computing centres in the mid-1970s. Growing dissatisfaction with its interface and the eventual withdrawal of Atlas Laboratory support prompted British funding bodies to commission a successor program. === Oxford Concordance Program === The Oxford Concordance Program (OCP) was designed and written in FORTRAN by Susan Hockey and Ian Marriott at Oxford University Computing Services (OUCS) between 1979 and 1980 and first released in 1981. Hockey and Marriott acknowledged that OCP owed much to COCOA and the CLOC system at the University of Birmingham. OCP accepted COCOA-format markup to encode metadata such as author, act, scene, and line number, and was described by its authors as "a machine-independent text analysis program for producing word lists, indices and concordances in a variety of languages and alphabets." By the mid-1980s it had been licensed to approximately 240 institutions in 23 countries. A personal computer version, Micro-OCP, was developed for the IBM PC and sold by Oxford University Press from the late 1980s. Version 2 was rewritten in 1985–86 and documented in the same 1987 article by Hockey and co-author John Martin. === Personal computer era === The availability of affordable personal computers in the 1980s and 1990s enabled standalone concordancing applications that analysts could run locally without specialist computing facilities. MicroConcord, developed by Mike Scott and Tim Johns and published by Oxford University Press in 1993 for MS-DOS, was among the first concordancers designed specifically for classroom language teaching. WordSmith Tools, also developed by Mike Scott, was first released in 1996 and became one of the most widely used corpus analysis suites in academic linguistics research. Other tools from this era include TACT (University of Toronto, 1989), a suite of MS-DOS freeware programs for literary text analysis, and MonoConc, a Windows concordancer created by Michael Barlow. === Web-based concordancers === From the late 1990s onwards, web-based concordancers hosted on remote servers gave researchers browser access to large preloaded corpora without requiring local storage or processing. The Sketch Engine, developed by Adam Kilgarriff and Pavel Rychlý (Masaryk University), was launched commercially in July 2003 by Lexical Computing Limited and introduced word sketches—automatically generated one-page profiles of a word's typical grammatical relations and collocations. AntConc, created by Laurence Anthony at Waseda University, Tokyo, was first released in 2002 as freeware for Windows, macOS, and Linux. == Features == Modern concordancers typically offer a range of analytical functions beyond basic KWIC display. These commonly include: KWIC display with the node word centred and context words in aligned columns, sortable by the word one, two, or three positions to the left or right of the node (L1–L3 and R1–R3) Concordance plots, visualising the distribution of hits as marks along a scaled bar representing each text in the corpus Frequency and word lists, both alphabetical and ranked by frequency Collocation statistics, identifying words that co-occur with the search term more often than chance, quantified by measures such as mutual information, the t-score, or log-likelihood Keyword analysis, comparing word frequencies between a study corpus and a reference corpus to identify statistically distinctive items N-gram analysis, finding frequently recurring word sequences of a specified length Part-of-speech tagging integration, allowing searches filtered to particular grammatical categories Unicode support for multilingual text Bilingual and parallel concordancers additionally display aligned text in two or more languages side by side, enabling comparison of translation equivalents across language pairs. == Notable concordancers == === WordSmith Tools === Created by Mike Scott and first released in 1996, WordSmith Tools is a Windows corpus analysis suite that evolved from MicroConcord. Its three core modules are Concord (KWIC concordances), WordList (frequency and alphabetical word lists), and Keywords (statistical keyword identification relative to a reference corpus). Oxford University Press used WordSmith Tools for dictionary preparation work. Version 4.0 is freely available; later versions are sold by Lexical Analysis Software Limited. === AntConc === AntConc is a freeware, multiplatform concordancing toolkit created by Laurence Anthony, Professor of Applied Linguistics at Waseda University, Tokyo. First released in 2002 and formally described in a 2005 academic paper, it runs on Windows, macOS, and Linux. Its tools include a KWIC concordancer, a concordance plot for visualising distribution across texts, a collocates tool, a keyword list, and an n-gram analysis module. Because it is free and requires only plain text files, AntConc is widely used in linguistics courses and independent research worldwide. === Sketch Engine === The Sketch Engine is a corpus management and query system co-created by Adam Kilgarriff and Pavel Rychlý and launched in 2003 by Lexical Computing Limited. It provides browser-based access to over 800 corpora in more than 100 languages. Beyond concordance searching, it offers word sketches, collocation analysis, distributional thesaurus construction, keyword and terminology extraction, and diachronic analysis. It is used by major publishers including Macmillan and Oxford University Press for lexicographic research. A subset tool, SKELL (Sketch Engine for Language Learning), is freely accessible to individual learners. === Wmatrix === Wmatrix is a web-based corpus processing environment developed by Paul Rayson at the University Centre for Computer Corpus Research on Language (UCREL), Lancaster University. Alongside concordances and frequency lists, Wmatrix integrates CLAWS part-of-speech tagging and the USAS semantic tagger, enabling keyword analysis simultane

Vulnerability Discovery Model

A Vulnerability Discovery Model (VDM) uses discovery event data with software reliability models for predicting the same. A thorough presentation of VDM techniques is available in. Numerous model implementations are available in the MCMCBayes open source repository. Several VDM examples include: Alhazmi-Malaiya: Time based model (Alhazmi-Malaiya Logistic (AML) model) Alhazmi-Malaiya: Effort based model Rescorla: Quadratic Model and Exponential Model Anderson: Thermodynamic Model Kim: Weibull Model Linear Model Hump-Shaped Model Independent and Dependent Model Vulnerability Discovery Modeling using Bayesian model averaging Multivariate Vulnerability Discovery Models

NRENum.net

The NRENum.net service is an end-user ENUM service run by TERENA and the participating national research and education networking organisations (NRENs), primarily for academia. NRENum.net is considered as a complementary service and a valid alternative to the Golden ENUM tree. The domain nrenum.net is being populated in order to provide the infrastructure in DNS for storage of E.164 numbers. The NRENum.net service includes the operation of the Tier-0 root Domain Name Server(s) and the delegation of county codes to NRENum.net Registries. NRENum.net is a registered community trademark of TERENA. == Service description == E.164 Telephone Number Mapping (ENUM) is a standard protocol that is the result of work of the Internet Engineering Task Force's Telephone Number Mapping working group. ENUM translates a telephone number into a domain name. This allows users to continue to use the existing phone number formats they are familiar with, while allowing the call to be routed using DNS. This makes ENUM a quick, stable and cheap link between telecommunications systems and the Internet. RFC 3761 discusses the use of the Domain Name System for storage of E.164 numbers. More specifically, how DNS can be used for identifying available services connected to one E.164 number. The RIPE NCC provides DNS operations for e164.arpa (known as Golden ENUM tree) in accordance with the instructions from the Internet Architecture Board. The NRENum.net service is an end-user ENUM service run by TERENA and the participating NRENs primarily for academia. NRENum.net is considered as a complementary service and a valid alternative to the Golden ENUM tree. The domain nrenum.net is being populated in order to provide the infrastructure in DNS for storage of E.164 numbers. The NRENum.net service includes the operation of the Tier-0 root Domain Name Servers and the delegation of county codes to NRENum.net Registries. NRENum.net is a registered community trademark of TERENA. NRENum.net facilitates services such as Voice over IP and videoconferencing. NRENum.net tree refers to the tree structure where: Tier-0 root Domain Name Servers (technically one master and several secondary servers ensuring resilience) are run by the hosting organisations and coordinated by the NRENum.net Operations Team. Tier-1 Domain Name Servers are run by the NRENum.net (national or regional) Registries responsible for the country code(s) delegated. Tier-2 and lower DNS sub-delegations may be implemented, regulated by the national service policies. An NRENum.net Registry is an entity that is authorised by the NRENum.net Operations Team to operate the national or regional Tier-1 Domain Name Server and be responsible for the county code(s) delegated. In many countries there is a National Research and Education Networking organisation (NREN) that acts as the Registry of the country. An NRENum.net Registrar is responsible for the number/block registration in the Tier-1 DNS and a Number Validation Entity is responsible for the validation of the E.164 telephone numbers to be registered. The NREN may at the same time have the role of the NRENum.net Registry, Registrar and Validation Entity for the country code(s) delegated. A Registrant (end user) is an E.164 telephone number holder. Holders of E.164 numbers who want to be listed in the service must contact the appropriate NRENum.net Registrar. Number (block) delegation is the technical process of assigning country codes to national registries, or number blocks under country codes to end users. Number (block) registration is the technical process of configuring DNS and populating it with the appropriate ENUM records (i.e., adding NAPTR records to DNS) via registrars. The ITU-T strictly regulates the number structure of valid E.164 telephone numbers and assigns number blocks to national authorities (telecom regulators) or recently to global entities directly. The national authorities can further delegate the number ranges to local operators within the country or region. A virtual number has either a non-valid E.164 number structure (e.g., longer than 15 digits) or has a valid structure but is not assigned to any national authorities or operators. The number Validation Entity is responsible for checking the numbers to be registered to NRENum.net. == History == The idea for the NRENum.net service was conceived in 2006. NRENum.net became operational in August 2006, and was run by Bernie Höneisen, a staff member of SWITCH, and Kewin Stöckigt, a staff member of AARNet, as a private service, with technical support from SWITCH and the participants in the TERENA Task Force on Enhanced Communication Services (TF-ECS). When that task force completed its activities in 2008, TERENA agreed to take over the coordination of the NRENum.net service. By that time, nine NRENs had joined NRENum.net. The service continued to grow during the next years, and in March 2012 NRENum.net went global when RNP from Brazil joined the service as its 14th partificpant and the first outside Europe. In 2011, the participants decided to migrate the operation of the service's master Domain Name Server to NIIF and the operation of the two secondary DNSs to CARNET and SWITCH. In 2013, Internet2, AARNet and NORDUnet set up additional secondary Domain Name Servers for their regions, thereby completing the global distribution of DNS slaves and bringing the resilience of the NRENum.net infrastructure to a high level. == Governance == TERENA has established a lightweight global governance structure. The Global NRENum.net Governance Committee (GNGC) is the highest-level strategic body responsible for overall NRENum.net service definition, sustainability and long-term strategy. This includes formulating and recommending service governance principles and policies. Its members are nominated by the NRENum.net Registries in the various world regions, and are appointed by TERENA. The GNGC is composed of two members representing Europe, two representing the Asia-Pacific region, and two representing the Americas. The NRENum.net Operations Team is responsible for the day-to-day operations of the Tier-0 root DNSs and the handling of country code delegation requests. It may escalate technical or policy issues to the GNGC for discussion. TERENA is responsible for ensuring the correct and secure operations of the NRENum.net service performed by the NRENum.net Operations Team and governance by the GNGC. TERENA also supports the development of technical improvements to the NRENum.net service and promotes the deployment of NRENum.net worldwide. == Geographical deployment == Thirty-two county codes are delegated in the NRENum.net service. Below these are listed per world region. === Europe === === Asia-Pacific === === North America === +1 United States (Internet2) === Latin America === === Caribbean === === Africa === +262 Réunion, Mayotte (RENATER)

Hybrid argument (cryptography)

In cryptography, the hybrid argument is a proof technique used to show that two distributions are computationally indistinguishable. == History == Hybrid arguments had their origin in a papers by Andrew Yao in 1982 and Shafi Goldwasser and Silvio Micali in 1983. == Formal description == Formally, to show two distributions D1 and D2 are computationally indistinguishable, we can define a sequence of hybrid distributions D1 := H0, H1, ..., Ht =: D2 where t is polynomial in the security parameter n. Define the advantage of any probabilistic efficient (polynomial-bounded time) algorithm A as A d v H i , H i + 1 d i s t ( A ) := | Pr [ x ← $ H i : A ( x ) = 1 ] − Pr [ x ← $ H i + 1 : A ( x ) = 1 ] | , {\displaystyle {\mathsf {Adv}}_{H_{i},H_{i+1}}^{\mathsf {dist}}(\mathbf {A} ):=\left|\Pr[x{\stackrel {\$}{\gets }}H_{i}:\mathbf {A} (x)=1]-\Pr[x{\stackrel {\$}{\gets }}H_{i+1}:\mathbf {A} (x)=1]\right|,} where the dollar symbol ($) denotes that we sample an element from the distribution at random. By triangle inequality, it is clear that for any probabilistic polynomial time algorithm A, A d v D 1 , D 2 d i s t ( A ) ≤ ∑ i = 0 t − 1 A d v H i , H i + 1 d i s t ( A ) . {\displaystyle {\mathsf {Adv}}_{D_{1},D_{2}}^{\mathsf {dist}}(\mathbf {A} )\leq \sum _{i=0}^{t-1}{\mathsf {Adv}}_{H_{i},H_{i+1}}^{\mathsf {dist}}(\mathbf {A} ).} Thus there must exist some k s.t. 0 ≤ k < t(n) and A d v H k , H k + 1 d i s t ( A ) ≥ A d v D 1 , D 2 d i s t ( A ) / t ( n ) . {\displaystyle {\mathsf {Adv}}_{H_{k},H_{k+1}}^{\mathsf {dist}}(\mathbf {A} )\geq {\mathsf {Adv}}_{D_{1},D_{2}}^{\mathsf {dist}}(\mathbf {A} )/t(n).} Since t is polynomial-bounded, for any such algorithm A, if we can show that it has a fixed negligible advantage function ε(n) between distributions Hi and Hi+1 for every i, so in particular, ϵ ( n ) ≥ A d v H k , H k + 1 d i s t ( A ) ≥ A d v D 1 , D 2 d i s t ( A ) / t ( n ) , {\displaystyle \epsilon (n)\geq {\mathsf {Adv}}_{H_{k},H_{k+1}}^{\mathsf {dist}}(\mathbf {A} )\geq {\mathsf {Adv}}_{D_{1},D_{2}}^{\mathsf {dist}}(\mathbf {A} )/t(n),} then it immediately follows that its advantage to distinguish the distributions D1 = H0 and D2 = Ht must also be negligible. == Applications == The hybrid argument is extensively used in cryptography. Some simple proofs using hybrid arguments are: If one cannot efficiently predict the next bit of the output of some number generator, then this generator is a pseudorandom number generator (PRG). We can securely expand a PRG with 1-bit output into a PRG with n-bit output.

Forking lemma

The forking lemma is any of a number of related lemmas in cryptography research. The lemma states that if an adversary (typically a probabilistic Turing machine), on inputs drawn from some distribution, produces an output that has some property with non-negligible probability, then with non-negligible probability, if the adversary is re-run on new inputs but with the same random tape, its second output will also have the property. This concept was first used by David Pointcheval and Jacques Stern in "Security proofs for signature schemes," published in the proceedings of Eurocrypt 1996. In their paper, the forking lemma is specified in terms of an adversary that attacks a digital signature scheme instantiated in the random oracle model. They show that if an adversary can forge a signature with non-negligible probability, then there is a non-negligible probability that the same adversary with the same random tape can create a second forgery in an attack with a different random oracle. The forking lemma was later generalized by Mihir Bellare and Gregory Neven. The forking lemma has been used and further generalized to prove the security of a variety of digital signature schemes and other random-oracle based cryptographic constructions. == Statement of the lemma == The generalized version of the lemma is stated as follows. Let A be a probabilistic algorithm, with inputs (x, h1, ..., hq; r) that outputs a pair (J, y), where r refers to the random tape of A (that is, the random choices A will make). Suppose further that IG is a probability distribution from which x is drawn, and that H is a set of size h from which each of the hi values are drawn according to the uniform distribution. Let acc be the probability that on inputs distributed as described, the J output by A is greater than or equal to 1. We can then define a "forking algorithm" FA that proceeds as follows, on input x: Pick a random tape r for A. Pick h1, ..., hq uniformly from H. Run A on input (x, h1, ..., hq; r) to produce (J, y). If J = 0, then return (0, 0, 0). Pick h'J, ..., h'q uniformly from H. Run A on input (x, h1, ..., hJ−1, h'J, ..., h'q; r) to produce (J', y'). If J' = J and hJ ≠ h'J then return (1, y, y'), otherwise, return (0, 0, 0). Let frk be the probability that FA outputs a triple starting with 1, given an input x chosen randomly from IG. Then frk ≥ acc ⋅ ( acc q − 1 h ) . {\displaystyle {\text{frk}}\geq {\text{acc}}\cdot \left({\frac {\text{acc}}{q}}-{\frac {1}{h}}\right).} === Intuition === The idea here is to think of A as running two times in related executions, where the process "forks" at a certain point, when some but not all of the input has been examined. In the alternate version, the remaining inputs are re-generated but are generated in the normal way. The point at which the process forks may be something we only want to decide later, possibly based on the behavior of A the first time around: this is why the lemma statement chooses the branching point (J) based on the output of A. The requirement that hJ ≠ h'J is a technical one required by many uses of the lemma. (Note that since both hJ and h'J are chosen randomly from H, then if h is large, as is usually the case, the probability of the two values not being distinct is extremely small.) === Example === For example, let A be an algorithm for breaking a digital signature scheme in the random oracle model. Then x would be the public parameters (including the public key) A is attacking, and hi would be the output of the random oracle on its ith distinct input. The forking lemma is of use when it would be possible, given two different random signatures of the same message, to solve some underlying hard problem. An adversary that forges once, however, gives rise to one that forges twice on the same message with non-negligible probability through the forking lemma. When A attempts to forge on a message m, we consider the output of A to be (J, y) where y is the forgery, and J is such that m was the Jth unique query to the random oracle (it may be assumed that A will query m at some point, if A is to be successful with non-negligible probability). (If A outputs an incorrect forgery, we consider the output to be (0, y).) By the forking lemma, the probability (frk) of obtaining two good forgeries y and y' on the same message but with different random oracle outputs (that is, with hJ ≠ h'J) is non-negligible when acc is also non-negligible. This allows us to prove that if the underlying hard problem is indeed hard, then no adversary can forge signatures. This is the essence of the proof given by Pointcheval and Stern for a modified ElGamal signature scheme against an adaptive adversary. == Known issues with application of forking lemma == The reduction provided by the forking lemma is not tight. Pointcheval and Stern proposed security arguments for Digital Signatures and Blind Signature using Forking Lemma. Claus P. Schnorr provided an attack on blind Schnorr signatures schemes, with more than p o l y l o g ( n ) {\displaystyle polylog(n)} concurrent executions (the case studied and proven secure by Pointcheval and Stern). A polynomial-time attack, for Ω ( n ) {\displaystyle \Omega (n)} concurrent executions, was shown in 2020 by Benhamouda, Lepoint, Raykova, and Orrù. Schnorr also suggested enhancements for securing blind signatures schemes based on discrete logarithm problem.

Amazon Kinesis

Amazon Kinesis is a family of services provided by Amazon Web Services (AWS) for processing and analyzing real-time streaming data at a large scale. Launched in November 2013, it offers developers the ability to build applications that can consume and process data from multiple sources simultaneously. Kinesis supports multiple use cases, including real-time analytics, log and event data collection, and real-time processing of data generated by IoT devices. == History == Amazon Kinesis was launched by Amazon Web Services (AWS) in November 2013 as a managed service for processing and analyzing real-time streaming data at a large scale. The service was introduced to address the growing need for businesses to process and analyze data as it was generated, rather than in batches, allowing for real-time insights and decision-making. Since its launch, the Amazon Kinesis family of services has expanded to include four main components: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams. Each of these components serves a specific purpose in the processing and analysis of real-time streaming data. In August 2015, AWS announced the availability of Kinesis Data Firehose, a fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, and Amazon Elasticsearch. A year later in August 2016, AWS launched Kinesis Data Analytics, enabling customers to analyze streaming data in real time using standard SQL queries. AWS introduced Kinesis Video Streams, a fully managed service for securely capturing, processing, and storing video streams for analytics and machine learning applications, was introduced by AWS in November 2017. == Components == Amazon Kinesis is composed of four main services: Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, and Kinesis Video Streams. === Kinesis Data Streams === Kinesis Data Streams is a scalable and durable real-time data streaming service that captures and processes gigabytes of data per second from multiple sources. It enables the storage and processing of data in real time, making it useful for applications that require immediate insights, such as monitoring and alerting. === Kinesis Data Firehose === Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon Elasticsearch, and AWS-partner data stores. With Data Firehose, users can configure and scale data delivery without manual intervention. === Kinesis Data Analytics === Kinesis Data Analytics enables the analysis of streaming data in real time using standard SQL or Apache Flink. === Kinesis Video Streams === Kinesis Video Streams is a fully managed service for securely capturing, processing, and storing video streams for analytics and machine learning. It supports multiple video codecs and streaming protocols, making it suitable for various use cases, such as security and surveillance, video-enabled IoT devices, and live event broadcasting. == Integration == Amazon Kinesis can be easily integrated with other AWS services, such as AWS Lambda, Amazon S3, Amazon Redshift, and Amazon OpenSearch. This integration enables developers to build end-to-end streaming data processing applications, taking advantage of the extensive AWS ecosystem. == Use cases == Some common use cases for Amazon Kinesis include: Real-time analytics: Analyzing streaming data in real time to provide immediate insights and make data-driven decisions. Log and event data collection: Collecting, processing, and analyzing log and event data generated by applications, infrastructure, and devices. IoT data processing: Processing and analyzing large volumes of data generated by IoT devices in real time. Machine learning: Ingesting and processing video streams for machine learning applications, such as object recognition, facial recognition, and sentiment analysis. == Pricing == Amazon Kinesis follows a pay-as-you-go pricing model, with costs depending on the chosen service, data volume, and processing power required. AWS provides a free tier for Kinesis Data Streams and Kinesis Data Firehose, allowing users to get started with the services at no cost.

IWARP

iWARP is a computer networking protocol that implements remote direct memory access (RDMA) for efficient data transfer over Internet Protocol networks. Contrary to some accounts, iWARP is not an acronym. Because iWARP is layered on Internet Engineering Task Force (IETF)-standard congestion-aware protocols such as Transmission Control Protocol (TCP) and Stream Control Transmission Protocol (SCTP), it makes few requirements on the network, and can be successfully deployed in a broad range of environments. == History == In 2007, the IETF published five Request for Comments (RFCs) that define iWARP: RFC 5040 A Remote Direct Memory Access Protocol Specification is layered over Direct Data Placement Protocol (DDP). It defines how RDMA Send, Read, and Write operations are encoded using DDP into headers on the network. RFC 5041 Direct Data Placement over Reliable Transports is layered over MPA/TCP or SCTP. It defines how received data can be directly placed into an upper layer protocols receive buffer without intermediate buffers. RFC 5042 Direct Data Placement Protocol (DDP) / Remote Direct Memory Access Protocol (RDMAP) Security analyzes security issues related to iWARP DDP and RDMAP protocol layers. RFC 5043 Stream Control Transmission Protocol (SCTP) Direct Data Placement (DDP) Adaptation defines an adaptation layer that enables DDP over SCTP. RFC 5044 Marker PDU Aligned Framing for TCP Specification defines an adaptation layer that enables preservation of DDP-level protocol record boundaries layered over the TCP reliable connected byte stream. These RFCs are based on the RDMA Consortium's specifications for RDMA over TCP. The RDMA Consortium's specifications are influenced by earlier RDMA standards, including Virtual Interface Architecture (VIA) and InfiniBand (IB). Since 2007, the IETF has published three additional RFCs that maintain and extend iWARP: RFC 6580 IANA Registries for the Remote Direct Data Placement (RDDP) Protocols published in 2012 defines IANA registries for Remote Direct Data Placement (RDDP) error codes, operation codes, and function codes. RFC 6581 Enhanced Remote Direct Memory Access (RDMA) Connection Establishment published in 2011 fixes shortcomings with iWARP connection setup. RFC 7306 Remote Direct Memory Access (RDMA) Protocol Extensions published in 2014 extends RFC 5040 with atomic operations and RDMA Write with Immediate Data. == Protocol == The main component in the iWARP protocol is the Direct Data Placement Protocol (DDP), which permits the actual zero-copy transmission. DDP itself does not perform the transmission; the underlying protocol (TCP or SCTP) does. However, TCP does not respect message boundaries; it sends data as a sequence of bytes without regard to protocol data units (PDU). In this regard, DDP itself may be better suited for SCTP, and indeed the IETF proposed a standard RDMA over SCTP. To run DDP over TCP requires a tweak known as marker PDU aligned (MPA) framing to guarantee boundaries of messages. Furthermore, DDP is not intended to be accessed directly. Instead, a separate RDMA protocol (RDMAP) provides the services to read and write data. Therefore, the entire RDMA over TCP specification is really RDMAP over DDP over either MPA/TCP or SCTP. All of these protocols can be implemented in hardware. Unlike IB, iWARP only has reliable connected communication, as this is the only service that TCP and SCTP provide. The iWARP specification omits other features of IB, such as Send with Immediate Data operations. With RFC 7306, the IETF is working to reduce these omissions. == Implementation == Because a kernel implementation of the TCP stack can be seen as a bottleneck, the protocol is typically implemented in hardware RDMA network interface controllers (rNICs). As simple data losses are rare in tightly coupled network environments, the error-correction mechanisms of TCP may be performed by software while the more frequently performed communications are handled strictly by logic embedded on the rNIC. Similarly, connections are often established entirely by software and then handed off to the hardware. Furthermore, the handling of iWARP specific protocol details is typically isolated from the TCP implementation, allowing rNICs to be used for both as RDMA offload and TCP offload (in support of traditional sockets based TCP/IP applications). The portion of the hardware implementation used for implementing the TCP protocol is known as the TCP Offload Engine (TOE). TOE itself does not prevent copying on the reception side, and must be combined with RDMA hardware for zero-copy results. The RDMA / TCP specification is a set of different wire protocols intended to be implemented in hardware (though it seems feasible to emulate it in software for compatibility but without the performance benefits). == Interfaces == iWARP is a protocol, not an implementation, but defines protocol behavior in terms of the operations that are legal for the protocol, known as Verbs. As such, iWARP does not have any single standard programming interface. However, programming interfaces tend to very closely correspond to the Verbs. Several programmatic interfaces have been proposed, including OpenFabrics Verbs, Network Direct, uDAPL, kDAPL, IT-API, and RNICPI. Implementations of some of these interfaces are available for different platforms, including Windows and Linux. == Services available == Networking services implemented over iWARP include those offered in the OpenFabrics Enterprise Distribution (OFED) by the OpenFabrics Alliance for Linux operating systems, and by Microsoft Windows via Network Direct. NVMe over Fabrics (NVMEoF) iSCSI Extensions for RDMA (iSER) Server Message Block Direct (SMB Direct) Sockets Direct Protocol (SDP) SCSI RDMA Protocol (SRP) Network File System over RDMA (NFS over RDMA) GPUDirect