Skip to content

Compiling asn1 specifications

Jump to bottom
mitshell edited this page Feb 27, 2024 · 1 revision

How to compile ASN.1 specifications with the pycrate ASN.1 compiler

What is ASN.1

ASN.1 is a notation used to describe abstract structures.

Those are made from basic types:

  • NULL
  • BOOLEAN
  • INTEGER
  • REAL
  • ENUMERATED
  • BIT STRING
  • OCTET STRING
  • *String (many String types are defined, each using a different codec)
  • UTCTime and GeneralizedTime

from constructed types:

  • CHOICE
  • SEQUENCE and SET
  • SEQUENCE OF and SET OF

and from ASN.1 specific types:

  • OBJECT IDENTIFIER and RELATIVE-OID
  • EXTERNAL, EMBEDDED PDV and CHARACTER STRING

After structures are defined according to these types, encoders must be used to serialize data. This enables heterogeous systems to communicate in an unified way according to a given ASN.1 definition.

The different encoders defined are:

  • BER (Basic Encoding Rules), CER (Canonical Encoding Rules) and DER (Distinguished Encoding Rules): byte-oriented codecs
  • PER (Packed Encoding Rules), aligned and unaligned: bit-oriented codecs
  • XER (XML Encoding Rules) and GSER (Generic String Encoding Rules): text-oriented

The current specification for ASN.1 and its codecs can be found in ITU-T documents:

  • X.680, for the ASN.1 notation syntax
  • X.681, for specific CLASS objects
  • X.682, for specific constraints
  • X.683, for parameterization
  • X.690, for BER, CER and DER codecs
  • X.691, for PER codecs
  • X.693, for the XER codec
  • GSER codec is defined by IETF under the RFC 3641.

ASN.1 was originally specified in 1988 in the ITU-T recommendation X.208.

Good ressources about ASN.1 can be found in those two books that are available for free on the web:

  • ASN.1 - Communication Between Heterogeneous Systems, by Olivier Dubuisson and translated by Philippe Fouquart
  • ASN.1 Complete, by John Larmouth

Here are some other available useful ressources:

  • OSS Nokalva provides an online ASN.1 compiler, plus multiple encoders and decoders at the asn1-playground.
  • Lev Walkin proposes asn1c, a very complete ASN.1 to C and C++ compiler plus multiple encoders and decoders (and also an online one).
  • Fabrice Bellard proposes ffasn1, which has a free ASN.1 message converter and editor which supports many encoding rules too.
  • ITU-T has a complete web page referencing tools and softwares supporting ASN.1 at the ITU-T ASN.1 tools.
  • A very complete webpage at zytrax explaining the ASN.1 syntax in its entirety, the DER encoding, and how it is used for certificate encoding

Support in pycrate

The current support of the ASN.1 syntax in the pycrate ASN.1 compiler corresponds to the last ASN.1 versions, i.e. the 2015 one, which is itself quite similar to the ASN.1 standard from 2002. There are still few limitations in the current version of the compiler (see limitations).

In particular, the following general features are supported:

  • automatic and manual tagging
  • type extensibility
  • constraints: value and range, size, permitted alphabet
  • CLASS objects (also called information objects), open types and table constraints
  • parameterization
  • ANY type and the DEFINED BY construction (required for some older specifications)

The pycrate ASN.1 runtime supports the following encoders:

  • BER, CER and DER
  • PER aligned and unaligned
  • OER and COER
  • JER
  • ASN.1 textual value notation

and the following features:

  • table constraint resolution at runtime, hence resolving open types when possible
  • fragmentation in BER, CER and PER (DER does not use fragmentation)
  • runtime checking against value, range and size constraints (which can be disabled)

Limitations

There are still few limitations in the current code. On the compiler side:

  • the left part of an ASN.1 assignment and the assignment sign (::=) have to be on a single line
  • C-style comments between /* and */ are not supported
  • old-school ASN.1 MACROs are not supported, therefore the compiler cannot process SNMP MIBs
  • some extreme parameterization cases can lead to a compiler error (see the object AllPackagesAS in the ITU-T Q.775 specification)
  • recursive or circular objects definition together with parameterization are not supported
  • some constraints which do not have any impact on encoding rules are barely or not processed, this includes WITH COMPONENT (WITH COMPONENTS is however processed), PATTERN, SETTINGS and CONSTRAINED BY constraints
  • the constraint ENCODE BY is not processed either
  • new universal objects are not supported, this includes DATE, TIME-OF-DAY, DATE-TIME, DURATION, OID-IRI and RELATIVE-OID-IRI objects

All those cases are rarely found into classical ASN.1 specifications, and would require some more code to be supported. This is mostly why they are not supported, yet.

On the runtime side:

  • XER and any XML-related features are not supported
  • GSER and other encoding rules are not supported
  • ECN is not supported either
  • some codecs for specific String types are not supported (mainly because Python does not have support for them): TeletexString, VideotextString, GraphicString and GenericString, all using the ISO-2022 encoding
  • the runtime is not thread-safe: only a single thread must make use of an ASN.1 specification at a time, otherwise, encoding / decoding values can get mixed

How to

In order to work with an ASN.1-specified protocol or data format, here are the two steps to follow:

  • compile the ASN.1 definition into Python source code
  • use the generated Python code to encode specific values that you assign to given objects, and decode bytes' buffers to specific values

In order to compile an ASN.1 specification, few possibilities are offered.

the tool pycrate_asn1compile.py

A specific tool tools/pycrate_asn1compile.py is provided. Just requests its help to see how it works:

usage: pycrate_asn1compile.py [-h] [-s SPEC] [-i INPUT [INPUT ...]]
                              [-o OUTPUT] [-j] [-fautotags] [-fextimpl]
                              [-fverifwarn]

compile ASN.1 input file(s) for the pycrate ASN.1 runtime

optional arguments:
  -h, --help            show this help message and exit
  -s SPEC               provide a specification shortname, instead of ASN.1
                        input file(s)
  -i INPUT [INPUT ...]  ASN.1 input file(s) or directory
  -o OUTPUT             compiled output Python (and json) source file(s)
  -j                    output a json file with information on ASN.1 objects
                        dependency
  -fautotags            force AUTOMATIC TAGS for all ASN.1 modules
  -fextimpl             force EXTENSIBILITY IMPLIED for all ASN.1 modules
  -fverifwarn           force warning instead of raising during the
                        verification stage

It takes a directory containing ASN.1 definitions (file with .asn[1] suffix), or one or multiple files as input, and generates a Python source code as output. It can also generate a json file detailing ASN.1 objects' dependency, with the -j option. Few more options starting with -f' are available to be passed to the compiler, in order to force specific behaviours.

Here is an example with the X2AP protocol ASN.1 definition:

$ ./tools/pycrate_asn1compile.py -i ./pycrate_asn1dir/3GPP_EUTRAN_X2AP_36423/ -o x2ap -j
[proc] module X2AP-Containers (oid: [0, 4, 0, 0, 21, 3, 2, 1, 5]): 16 ASN.1 assignments found
[proc] module X2AP-CommonDataTypes (oid: [0, 4, 0, 0, 21, 3, 2, 1, 3]): 10 ASN.1 assignments found
[proc] module X2AP-IEs (oid: [0, 4, 0, 0, 21, 3, 2, 1, 2]): 342 ASN.1 assignments found
[proc] module X2AP-PDU-Contents (oid: [0, 4, 0, 0, 21, 3, 2, 1, 1]): 250 ASN.1 assignments found
[proc] module X2AP-Constants (oid: [0, 4, 0, 0, 21, 3, 2, 1, 4]): 233 ASN.1 assignments found
[proc] module X2AP-PDU-Descriptions (oid: [0, 4, 0, 0, 21, 3, 2, 1, 0]): 36 ASN.1 assignments found
--- compilation cycle ---
--- compilation cycle ---
--- compilation cycle ---
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: ['X2AP-Containers', 'X2AP-CommonDataTypes', 'X2AP-IEs', 'X2AP-PDU-Contents', 'X2AP-Constants', 'X2AP-PDU-Descriptions']
[proc] ASN.1 objects compiled: 421 types, 198 sets, 262 values
[proc] done
$ ls -l x2ap*
-rw-rw-r-- 1 user user  255442 Oct 10 14:07 x2ap.json
-rw-rw-r-- 1 user user 1452734 Oct 10 14:07 x2ap.py

From here, the x2ap.py and x2ap.json can be used as explained in the wiki part on the ASN.1 runtime:

>>> import x2ap
>>> # do whatever encodings / decodings you want with this generated module

Another way to use the compiler is provided in order to recompile pycrate internal ASN.1 modules. It corresponds to the -s option: using it trigger a recompilation of the corresponding module within the pycrate_asn1dir sub-directory, producing both python and json output file:

$ ./tools/pycrate_asn1compile.py -s RANAP
[proc] starting with ASN.1 specification: 3GPP_UTRAN_RANAP_25413
[proc] [RANAP-CommonDataTypes.asn] module RANAP-CommonDataTypes (oid: [0, 4, 0, 0, 20, 3, 0, 1, 3]): 8 ASN.1 assignments found
[proc] [RANAP-Constants.asn] module RANAP-Constants (oid: [0, 4, 0, 0, 20, 3, 0, 1, 4]): 370 ASN.1 assignments found
[proc] [RANAP-Containers.asn] module RANAP-Containers (oid: [0, 4, 0, 0, 20, 3, 0, 1, 5]): 15 ASN.1 assignments found
[proc] [RANAP-IEs.asn] module RANAP-IEs (oid: [0, 4, 0, 0, 20, 3, 0, 1, 2]): 507 ASN.1 assignments found
[proc] [RANAP-PDU-Contents.asn] module RANAP-PDU-Contents (oid: [0, 4, 0, 0, 20, 3, 0, 1, 1]): 401 ASN.1 assignments found
[proc] [RANAP-PDU-Descriptions.asn] module RANAP-PDU-Descriptions (oid: [0, 4, 0, 0, 20, 3, 0, 1, 0]): 60 ASN.1 assignments found
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: [u'RANAP-CommonDataTypes', u'RANAP-Constants', u'RANAP-Containers', u'RANAP-IEs', u'RANAP-PDU-Contents', u'RANAP-PDU-Descriptions']
[proc] ASN.1 objects compiled: 601 types, 336 sets, 418 values
[proc] done
/home/user/python/pycrate_asn1dir/RANAP.py file created
/home/user/python/pycrate_asn1dir/RANAP.json file created

In case you have installed pycrate system-wide, you will certainly need to run the compiler as root, otherwise the produced files won't be allowed to be copied into your system directory containing pycrate.

the pycrate_asn1c modules

The ASN.1 compiler actually lies in the file pycrate_asn1c/asnproc.py. The function compile_text() takes a textual input containing ASN.1 module(s), or an iterable of textual inputs, and process them to build all the ASN.1 modules into a dictionnary in the global class called GLOBAL.

The compiler function can take specific keywords to force specific behaviours if required:

  • autotags=True
  • extimpl=True
  • verifwarn=True Those correspond to the behaviours described in the help string returned by the pycrate_asn1compile.py tool.

Here is an example with the ASN.1 test file provided in the test directory:

>>> from pycrate_asn1c.asnproc import *
>>> help(compile_text)
[...]
>>> asntxt = open('./test/res/Hardcore.asn').read()
>>> compile_text(asntxt)
[proc] module HardcoreSyntax (oid: []): 116 ASN.1 assignments found
--- compilation cycle ---
--- compilation cycle ---
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: ['HardcoreSyntax']
[proc] ASN.1 objects compiled: 75 types, 3 sets, 37 values
[proc] done
>>> list(GLOBAL.MOD.keys())
['_IMPL_', '_USER_', 'HardcoreSyntax']
>>> GLOBAL.MOD['HardcoreSyntax'] 
{
_name_: 'HardcoreSyntax',
_oidstr_: '',
_oid_: [],
_tag_: 'AUTOMATIC',
_ext_: None,
_exp_: None,

[...]

Recur1: <Recur1 (SEQUENCE)>,
Recur2: <Recur2 (SEQUENCE)>,
ASNWrapper: <ASNWrapper (SEQUENCE)>
}

In order to process a single ASN.1 definition corresponding to an ASN.1 object, most of the ASN.1 to Python translation is done by the ASN1Obj class in the file pycrate_asn1c/asnobj.py. This class has actually many methods, most of them being called to parse specific ASN.1 expressions and syntaxes.

After the GLOBAL class has been populated with Python objects built from the ASN.1 definition module(s) provided as input, the code generator must be called to generate a usable source file. Two different code generators are currently provided:

  • PycrateGenerator: generates one Python source file for the pycrate ASN.1 runtime
  • JSONDepGraphGenerator: generates one json file describing objects' dependencies

The function generate_modules() must be called with the code generator as 1st argument, and the destination file as 2nd argument. This will produce a source file at the given destination:

>>> generate_modules(PycrateGenerator, '/tmp/hardcore.py')
>>> generate_modules(JSONDepGraphGenerator, '/tmp/hardcore.json')

the pycrate_asn1dir directory

Many ready-to-use ASN.1 modules are provided in the pycrate_asn1dir/ directory. All ASN.1 definitions are placed in sub-directories, and corresponding Python and json source files are available directly there.

The links between those sub-directories and Python and json files are set in pycrate_asn1c/specdir.py.

It is possible to recompile a single ASN.1 specification, some of them, or all of them, with the generate_all() function in pycrate_asn1c/asnproc.py. This function takes a dictionnary as argument, linking destination name to sub-directory name. For instance, {'X2AP': '3GPP_EUTRAN_X2AP_36423', ...}. The default argument is the dictionnary ASN_SPECS from the specdir.py file.

So, it is possible to add a sub-directory in this pycrate_asn1dir directory, and a corresponding entry in the ASN_SPECS dictionnary. This will integrate the sub-directory as part of the default ASN.1 modules in the pycrate library.

Finally, it is possible to launch a recompilation of all ASN.1 modules, as set in the ASN_SPECS file by simply executing the asnproc.py Python file. This will call generate_all(ASN_SPECS) and regenerate all .py and .json files in pycrate_asn1dir:

$ python -m pycrate_asn1c.asnproc
[GEN] LPPa
[proc] starting with ASN.1 specification: 3GPP_EUTRAN_LPPa_36455
[proc] module LPPA-CommonDataTypes (oid: [0, 4, 0, 0, 21, 3, 6, 1, 3]): 11 ASN.1 assignments found
[proc] module LPPA-Constants (oid: [0, 4, 0, 0, 21, 3, 6, 1, 4]): 36 ASN.1 assignments found
[proc] module LPPA-Containers (oid: [0, 4, 0, 0, 21, 3, 6, 1, 5]): 16 ASN.1 assignments found
[proc] module LPPA-IEs (oid: [0, 4, 0, 0, 21, 3, 6, 1, 2]): 75 ASN.1 assignments found
[proc] module LPPA-PDU-Contents (oid: [0, 4, 0, 0, 21, 3, 6, 1, 1]): 35 ASN.1 assignments found
[proc] module LPPA-PDU-Descriptions (oid: [0, 4, 0, 0, 21, 3, 6, 1, 0]): 18 ASN.1 assignments found
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: [u'LPPA-CommonDataTypes', u'LPPA-Constants', u'LPPA-Containers', u'LPPA-IEs', u'LPPA-PDU-Contents', u'LPPA-PDU-Descriptions']
[proc] ASN.1 objects compiled: 106 types, 32 sets, 47 values
[proc] done
[GEN] TAP3
[proc] starting with ASN.1 specification: GSMA_TAP3_17102014
[proc] module TAP-0312 (oid: []): 329 ASN.1 assignments found
[proc] module RAP-0105 (oid: []): 44 ASN.1 assignments found
--- compilation cycle ---
--- verifications ---
[proc] ASN.1 modules processed: [u'TAP-0312', u'RAP-0105']
[proc] ASN.1 objects compiled: 371 types, 0 sets, 0 values
[proc] done
[...]

Working on compiled ASN.1 modules

It can be interesting to work on Python objects built in the GLOBAL.MOD dict just after a compilation stage, instead of working with the Python module generated. Those Python objects contain extended information, compared to the objects in the Python module generated:

  • all ASN.1 objects compiled have a _ref attribute which lists all other ASN.1 objects referenced by them
  • set of values are organized in root and ext lists, instead of specific ASN1Set objects
  • parameterized objects are defined with specific references to their formal parameters; moreover, they have a _param attribute which lists their parameters and corresponding pathes into themselves

Using the HardcoreSyntax.asn module already compiled, we can illustrate this:

>>> Mod = GLOBAL.MOD['HardcoreSyntax']
>>> Mod['Seq2A']._ref # listing all references to other ASN.1 objects
{ASN1RefSet(HardcoreSyntax.Test3),
 ASN1RefClassField(HardcoreSyntax.TEST3.&bool),
 ASN1RefClassField(HardcoreSyntax.TEST3.&index),
 ASN1RefClassField(HardcoreSyntax.TEST3.&Type)}
>>> Mod['INT02A']._param # listing formal parameters and corresponding pathes
{
tag: {'ref': [['tag', 0]], 'gov': <tag (INTEGER):  >},
st: {'ref': [['cont', 'first']], 'gov': <st (INTEGER):  >},
nd: {'ref': [['cont', 'second']], 'gov': <nd (INTEGER):  >},
lo: {'ref': [['const', 0, 'root', 0, 'lb']], 'gov': <lo (INTEGER):  >},
hi: {'ref': [['const', 0, 'root', 0, 'ub']], 'gov': <hi (INTEGER):  >}
}
>>> Mod['Test3']._val # listing root and extended parts of a set of values
{'ext': None,
 'root': [{
index: 1,
bool: True,
Type: <Type ([Real000] REAL)>
},

[...]

  {
index: 6,
bool: False,
Type: <Type ([Seq000] SEQUENCE)>
}]}

Last but not least (but not very usable with real-world specification, too :), the json file generated by the compiler can be visualized in the browser thanks to this D3-based javascript. So you can (try to) visualize all the dependencies between ASN.1 objects from a compiled specification !

Clone this wiki locally