CmdID / Accuracy
Accuracy
How often CmdID reads a real command line or configuration as what it is, per platform. The lines are taken from public corpora: reference examples, tldr-pages and Batfish's test configurations, none of them the files an index is built from. Measured on 2026-11-16, with the engine at commit 6121af974a42: 37,321 lines, 97.3% of them detected.
Scores
Per platform, and per corpus under it. A share is rounded down to a tenth of a per cent. The measures are explained below.
| Corpus | Lines | Detected | Exact | Described |
|---|---|---|---|---|
| aws | 7,398 | 99.5%7,363 | ||
aws-examplea command line from aws-cli's examples (awscli/examples) | 7,069 | 99.8%7,057 | 99.1%7,010 | 97.5%8,161 of 8,364 options |
tldr-awsan aws line from tldr-pages | 329 | 93.0%306 | - | 91.8%381 of 415 options |
| az | 6,587 | 99.7%6,573 | ||
az-examplethe first line of an example in an Azure CLI command's help | 6,395 | 99.9%6,392 | 99.6%6,370 | 91.7%20,534 of 22,392 options |
tldr-azan az line from tldr-pages | 192 | 94.2%181 | - | 82.6%324 of 392 options |
| docker | 1,002 | 99.6%998 | ||
docker-examplea command line from docker's command reference | 739 | 99.5%736 | - | 98.9%687 of 694 options |
tldr-dockera docker line from tldr-pages | 263 | 99.6%262 | - | 99.4%181 of 182 options |
| kubectl | 604 | 99.8%603 | ||
kubectl-examplea command line from the kubectl reference's examples | 381 | 100.0%381 | - | 100.0%457 of 457 options |
tldr-kubectla kubectl line from tldr-pages | 223 | 99.5%222 | - | 100.0%190 of 190 options |
| gcloud | 92 | 98.9%91 | ||
tldr-gclouda gcloud line from tldr-pages | 92 | 98.9%91 | - | 100.0%43 of 43 options |
| helm | 15 | 93.3%14 | ||
tldr-helma helm line from tldr-pages | 15 | 93.3%14 | - | 100.0%4 of 4 options |
| terraform | 105 | 100.0%105 | ||
tldr-terraforma terraform line from tldr-pages | 105 | 100.0%105 | - | 89.0%57 of 64 options |
| vault | 7 | 100.0%7 | ||
tldr-vaulta vault line from tldr-pages | 7 | 100.0%7 | - | 100.0%1 of 1 options |
| openssl | 50 | 98.0%49 | ||
tldr-opensslan openssl line from tldr-pages | 50 | 98.0%49 | - | 100.0%107 of 107 options |
| git | 775 | 98.9%767 | ||
tldr-gita git line from tldr-pages | 775 | 98.9%767 | - | 92.5%437 of 472 options |
| npm | 283 | 100.0%283 | ||
tldr-npman npm line from tldr-pages | 283 | 100.0%283 | - | 92.3%108 of 117 options |
| pip | 84 | 100.0%84 | ||
tldr-pipa pip line from tldr-pages | 84 | 100.0%84 | - | 100.0%52 of 52 options |
| ansible | 42 | 100.0%42 | ||
tldr-ansiblean ansible line from tldr-pages (ansible, ansible-playbook, ansible-galaxy ...) | 42 | 100.0%42 | - | 90.0%45 of 50 options |
| PowerShell | 9,686 | 99.7%9,659 | ||
ps-winserveran example line that calls its page's cmdlet, up to two per page | 4,389 | 99.9%4,386 | - | 96.2%7,539 of 7,832 parameters |
ps-officean example line that calls its page's cmdlet, up to two per page | 2,856 | 99.1%2,832 | - | 97.5%4,659 of 4,778 parameters |
ps-graphan example line that calls its page's cmdlet, up to two per page | 2,441 | 100.0%2,441 | - | 98.7%3,888 of 3,936 parameters |
| Junos | 5,056 | 98.6%4,989 | ||
junos-linea set line of a Junos test configuration | 5,030 | 98.6%4,964 | 96.4%4,849 | 93.0%21,186 of 22,768 configuration lines |
junos-curlya Junos test configuration in the { } form, whole | 26 | 96.1%25 | 92.3%24 | 94.1%415 of 441 configuration lines |
| Cisco IOS XE | 1,267 | 79.5%1,008 | ||
ios-stanzaa top-level line of a Cisco test configuration, with the lines under it, from a file not naming an OS other than IOS or IOS XE | 1,085 | 79.0%858 | - | 31.5%919 of 2,913 configuration lines |
ios-napalma top-level line of a configuration in napalm's IOS tests, with the lines under it | 73 | 86.3%63 | - | 33.3%66 of 198 configuration lines |
ios-ansiblea top-level line of a cisco.ios unit-test configuration, with the lines under it | 109 | 79.8%87 | - | 49.2%163 of 331 configuration lines |
Before, ios-stanza-allevery Cisco test configuration read, as scored before the split by OS | 1,085 | 79.0%858 | ||
| FortiOS | 310 | 99.6%309 | ||
fortios-blocka top-level config ... end block of a FortiOS test configuration | 310 | 99.6%309 | - | 99.2%1,125 of 1,134 configuration lines |
| PAN-OS | 2,796 | 99.4%2,780 | ||
panos-linea set line of a PAN-OS test configuration | 2,774 | 99.4%2,758 | 97.6%2,709 | 53.2%6,597 of 12,387 configuration lines |
panos-curlya PAN-OS test configuration in the { } form, whole | 22 | 100.0%22 | 86.3%19 | 70.9%630 of 888 configuration lines |
| Arista EOS | 1,162 | 51.2%595 | ||
eos-stanzaa top-level line of an Arista EOS test configuration, with the lines under it | 985 | 53.5%527 | - | 28.6%1,168 of 4,076 configuration lines |
eos-napalma top-level line of a configuration in napalm's EOS tests, with the lines under it | 41 | 39.0%16 | - | 51.4%73 of 142 configuration lines |
eos-ansiblea top-level line of an arista.eos unit-test configuration, with the lines under it | 136 | 38.2%52 | - | 41.3%81 of 196 configuration lines |
| Cisco IOS XE, whole configurations | 206 | 97.5%201 | ||
ios-filea Cisco test configuration, whole, from a file not naming an OS other than IOS or IOS XE | 170 | 97.0%165 | - | 33.7%1,082 of 3,203 configuration lines |
ios-napalm-filea configuration in napalm's IOS tests, whole | 17 | 100.0%17 | - | 46.4%304 of 655 configuration lines |
ios-ansible-filea cisco.ios unit-test configuration, whole | 19 | 100.0%19 | - | 46.7%168 of 359 configuration lines |
| Arista EOS, whole configurations | 176 | 85.7%151 | ||
eos-filean Arista EOS test configuration, whole | 120 | 92.5%111 | - | 31.0%1,502 of 4,841 configuration lines |
eos-napalm-filea configuration in napalm's EOS tests, whole | 17 | 82.3%14 | - | 48.5%103 of 212 configuration lines |
eos-ansible-filean arista.eos unit-test configuration, whole | 39 | 66.6%26 | - | 39.3%141 of 358 configuration lines |
| All platforms | 37,321 | 97.3%36,319 |
A row headed Before is the platform's score as it was counted before a change to its corpus, kept beside the new one for one release.
What each device set was read as
Per device set, how many of its lines each device reader claimed (the set's own reader's count in bold), how many were read as no device - a program, a cmdlet or something else - and how many nothing was found for. The Cisco set is split by what each file says its OS is: its name (ios-xe-...), a RANCID header naming a type, an ASA or PIX version line, or a version line of a release IOS XE has and IOS never had; a file that says none of these is IOS or IOS XE, not said which, and no OS is told from how a stanza looks. Batfish keeps its ASA, IOS XR and NX-OS test configurations in folders of their own, which are not read.
| Set | Lines | FortiOS | Cisco IOS XE CLI | Junos CLI | PAN-OS CLI | Arista EOS CLI | a program | nothing found |
|---|---|---|---|---|---|---|---|---|
junos-line | 5,030 | 0 | 0 | 4,964 | 0 | 0 | 66 | 0 |
junos-curly | 26 | 0 | 0 | 25 | 0 | 0 | 0 | 1 |
ios-stanzafiles naming IOS XE | 38 | 0 | 25 | 0 | 0 | 0 | 13 | 0 |
ios-stanzaIOS or IOS XE, the file not saying which | 1,047 | 0 | 833 | 0 | 0 | 7 | 202 | 5 |
fortios-block | 310 | 309 | 0 | 0 | 0 | 0 | 1 | 0 |
panos-line | 2,774 | 0 | 0 | 0 | 2,758 | 0 | 16 | 0 |
panos-curly | 22 | 0 | 0 | 0 | 22 | 0 | 0 | 0 |
eos-stanza | 985 | 0 | 277 | 0 | 0 | 527 | 160 | 21 |
ios-napalm | 73 | 0 | 63 | 0 | 0 | 0 | 8 | 2 |
ios-ansible | 109 | 0 | 87 | 0 | 0 | 5 | 7 | 10 |
eos-napalm | 41 | 0 | 21 | 0 | 0 | 16 | 2 | 2 |
eos-ansible | 136 | 0 | 69 | 0 | 0 | 52 | 9 | 6 |
ios-filefiles naming IOS XE | 7 | 0 | 7 | 0 | 0 | 0 | 0 | 0 |
ios-fileIOS or IOS XE, the file not saying which | 163 | 0 | 158 | 0 | 0 | 0 | 5 | 0 |
ios-napalm-file | 17 | 0 | 17 | 0 | 0 | 0 | 0 | 0 |
ios-ansible-file | 19 | 0 | 19 | 0 | 0 | 0 | 0 | 0 |
eos-file | 120 | 0 | 5 | 0 | 0 | 111 | 4 | 0 |
eos-napalm-file | 17 | 0 | 3 | 0 | 0 | 14 | 0 | 0 |
eos-ansible-file | 39 | 0 | 10 | 0 | 0 | 26 | 2 | 1 |
Left out of eos-napalm-file: 4 files (show_running_config___include_service_routing_protocols_model_multi_agent.text, show_running_config___include_service_routing_protocols_model_multi_agent.text, show_running_config___include_service_routing_protocols_model_multi_agent.text, show_running_config___section_router_bgp.text): no line of configuration.
Left out of ios-file: 1 file (ios-unicode): NUL bytes and no byte-order mark: an encoding the file does not say.
Left out of ios-stanza: 1 file (ios-unicode): NUL bytes and no byte-order mark: an encoding the file does not say.
Left out of junos-line, junos-curly: 1 file (nested-config-with-multiline-comment): no set line and no { } block.
Everyday shell lines
30,238 everyday shell lines from tldr-pages/tldr (its Linux, macOS, Windows and common pages, without those of the programs scored above: aws, az, docker, kubectl, gcloud, helm, terraform, vault, openssl, git, npm, pip and ansible). None of them is a network device's configuration; each one a device reader claims is a false positive.
| Read as | Lines |
|---|---|
| NetScaler ADC CLI | 0 |
| FortiOS | 0 |
| Cisco IOS XE CLI | 0 |
| Junos CLI | 0 |
| PAN-OS CLI | 0 |
| RouterOS CLI | 0 |
| Arista EOS CLI | 0 |
What is measured, and how
tools/eval/run.sh in CmdID's repository fetches each corpus at the commit listed below, takes its lines (tools/eval/corpus.py), runs the identifier on each, as the site's engine does, and counts (tools/eval/score.py). The same run writes the file this page is built from.
- Detected
- The share of lines the identifier read as what they are. What that means differs by platform:
- aws: a service and an operation found
- az: a command found, not a group
- docker: a command found, not only the group above an unknown word
- kubectl: a command found, not only the group above an unknown word
- gcloud: a command found, not only the group above an unknown word
- helm: a command found, not only the group above an unknown word
- terraform: a command found, not only the group above an unknown word
- vault: a command found, not only the group above an unknown word
- openssl: a command found
- git: a command found (git is read from tldr-pages)
- npm: a command found, not only the group above an unknown word
- pip: a command found, not only the group above an unknown word
- ansible: a command found, or a program that is its own command
- PowerShell: the page's cmdlet described
- Junos: read as Junos
- Cisco IOS XE: read as IOS XE
- FortiOS: read as FortiOS
- PAN-OS: read as PAN-OS
- Arista EOS: read as Arista EOS
- Cisco IOS XE, whole configurations: read as IOS XE, each file pasted whole
- Arista EOS, whole configurations: read as Arista EOS, each file pasted whole
- Exact
- Where the source gives the answer, the share of lines read as that answer:
aws-example: the operation the example's file is named foraz-example: the command whose help it is injunos-line: every word of the line a place the schema hasjunos-curly: every word of the configuration a place the schema haspanos-line: every word of the line a place the tree haspanos-curly: every word of the configuration a place the tree has
- Described
- Of the options, parameters or configuration lines in the lines detected, the share the identifier gives a description from its reference for. A FortiOS
editline names an object and is not counted.
What the scores do not measure
- Whether a description is the right one: described counts that an option has its reference's description, not that a person checked it fits the line.
- The Sigma rules a line trips, the LOLBAS and GTFOBins abuse it matches, its ATT&CK techniques, and decoded or deobfuscated scripts.
- Linux, macOS and Windows commands' options: everyday shell lines are used here only as lines a device reader must leave alone. NetScaler has no corpus of its own.
- Command lines from real hosts or logs, or written by intruders: the corpora are reference examples, tldr-pages' examples and parser test configurations.
Corpora
Only counts are shown on this site; no line of any corpus is. Each is read at the commit below, from the folders listed.
| Corpus | Commit | Licence | Read |
|---|---|---|---|
| batfish/batfish | 57ae1956686b2026-11-12 | Apache License 2.0 | projects/batfish/src/test/resources/org/batfish/grammar/juniper, projects/batfish/src/test/resources/org/batfish/grammar/cisco, projects/batfish/src/test/resources/org/batfish/grammar/fortios, projects/batfish/src/test/resources/org/batfish/grammar/palo_alto, projects/batfish/src/test/resources/org/batfish/vendor/arista/grammar |
| tldr-pages/tldr | 491149904dee2026-11-14 | CC BY 4.0 | pages |
| aws/aws-cli | f9ee5263749e2026-11-13 | Apache License 2.0 | awscli/examples |
| Azure/azure-cli | f280f6de0b992026-11-13 | MIT License | src/azure-cli/azure/cli/command_modules/*/_help.py, src/azure-cli/azure/cli/command_modules/*/**/_help.py |
| docker/cli | fdd4b8e5da332026-11-13 | Apache License 2.0 | docs/reference/commandline |
| kubernetes/website | fe0b202984832026-11-13 | CC BY 4.0 | content/en/docs/reference/kubectl/generated |
| MicrosoftDocs/windows-powershell-docs | 2029b4982aec2026-11-12 | CC BY 4.0 | docset/winserver2025-ps |
| MicrosoftDocs/office-docs-powershell | 7cfde21b7f662026-11-13 | CC BY 4.0 | exchange/exchange-ps/ExchangePowerShell, teams/teams-ps/MicrosoftTeams |
| MicrosoftDocs/microsoftgraph-docs-powershell | ee4ec05b3c952026-10-02 | CC BY 4.0 | microsoftgraph/graph-powershell-1.0 |
| napalm-automation/napalm | 820a06b2069e2026-08-12 | Apache License 2.0 | test/eos, test/ios |
| ansible-collections/arista.eos | 485c00dbb0a82026-11-10 | GNU GPL 3.0 | tests/unit/modules/network/eos/fixtures |
| ansible-collections/cisco.ios | 9102b81cb31b2026-11-12 | GNU GPL 3.0 | tests/unit/modules/network/ios/fixtures |
The identifier's own sources and their licences: Sources and licences.