BlackSponge's Bloghttps://bksp.space/blog/2021-01-08T00:00:00-05:00Migrating my DNS server from Bind9 to Knot DNS2021-01-08T00:00:00-05:002021-01-08T00:00:00-05:00BlackSpongetag:bksp.space,2021-01-08:/blog/en/2021-01-08-migrating-my-dns-server-from-bind9-to-knot-dns.html<p>Knot DNS simple configuration break down and migration steps from Bind9.</p><p>I own and manage a few domain names, including the <code>bksp.space</code> zone containing the domain of this blog, and run and host their nameservers myself. Since the beginning I have always been using Bind9 as this is the tool I learnt first and then I have not really looked into others. First because I didn't really care, I mean it worked already, why would I changed? And then after spending some time <a href="https://www.digitalocean.com/community/tutorials/how-to-setup-dnssec-on-an-authoritative-bind-dns-server-2">configuring DNSSEC</a>, including signature rotation and such, using bash scripts I did not wanted to go through the same burden again.</p>
<p>Even though, for a while I came upond other authoritative DNS software but none of the really catched my eyes, I mean <a href="https://www.powerdns.com/">PowerDNS</a> seams nice feature wise but does not look that much simpler to configure and operate. But recently I come accross people on the Fediverse using <a href="knot-dns.cz/">Knot DNS</a> to host <a href="https://mutu-ns.net.eu.org/about/">their zones</a> and being some curious I decided to have a look. So Knot-DNS is developped and used by the <a href="http://nic.cz/">Czech NIC</a> and, compared to the other softwares mentionned, is rather recent, <a href="https://en.wikipedia.org/wiki/BIND">Bind</a> and PowerDNS came out in 1986 and 1999 respectvely while Knot-DNS has been initially released in 2011. What I really liked though is the simplicity of the configuration and the possibility to have the keys and zone signature management fully automated, without too much hassle.</p>
<p>I you want to know more about it I recommned you to visit the documentation and guides, now I will simply describe what I have done to migrate the zone.</p>
<h2>Knot configuration</h2>
<p>To install knot simply fetch it from you distribution repositories, I am running Archlinux so in my case it would be <code>pacman -Sy knot</code>. In my previous setup I had a directory <code>my.zone.tld.d</code> for each zone containing the keys and the zone file, for <code>bksp.space</code> it looked like this.</p>
<div class="highlight"><pre><span></span>/var/named/bksp.space.d/
├── Kbksp.space.+007+05001.key
├── Kbksp.space.+007+05001.private
├── Kbksp.space.+007+15557.key
├── Kbksp.space.+007+15557.private
├── bksp.space.zone
├── bksp.space.zone.signed
└── dsset-bksp.space.
</pre></div>
<p>While the keys will not be living their anymore but in Knot keystore I could go back to having a single file per zone but I will stick to that structure in case I want to do some includes in the future. So to migrate the zone I simply copy the folder to what will be Knot storage directory.</p>
<div class="highlight"><pre><span></span>$ mkdir -p /var/lib/knot/zones
$ cp -r /var/named/bksp.space.d /var/lib/knot/zones
$ chown -R knot:knot /var/lib/knot/zones
</pre></div>
<p>Then it's time for the configuration, while it may look like YAML, experience has taught that it is not really, the first new blocks are quite simple, I configure the server to use the <code>knot</code> user, default, to listen on my public IPs and then configure logging to use the syslog and change server log level to waring to avoid unnessary notices about TCP client disconnection. Finally as mentionned before the base storage directory will be <code>/var/lib/knot</code>, again the default on my distribution.</p>
<div class="highlight"><pre><span></span><span class="nt">server</span><span class="p">:</span>
<span class="nt">rundir</span><span class="p">:</span> <span class="s">"/run/knot"</span>
<span class="nt">user</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">knot:knot</span>
<span class="nt">listen</span><span class="p">:</span> <span class="p p-Indicator">[</span> <span class="nv">198.51.100.1@53</span><span class="p p-Indicator">,</span> <span class="nv">2001</span><span class="p p-Indicator">:</span><span class="nv">db8</span><span class="p p-Indicator">:</span><span class="nv">cafe</span><span class="p p-Indicator">:</span><span class="nv">bc68</span><span class="p p-Indicator">::</span><span class="nv">1@53</span> <span class="p p-Indicator">]</span>
<span class="nt">log</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="nt">target</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">syslog</span>
<span class="nt">zone</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">info</span>
<span class="nt">control</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">info</span>
<span class="nt">server</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">warning</span>
<span class="nt">database</span><span class="p">:</span>
<span class="nt">storage</span><span class="p">:</span> <span class="s">"/var/lib/knot"</span>
</pre></div>
<p>Then I created a default policy for <a href="https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions">DNSSEC</a> management. I am using <code>rsasha1-nsec3-sha1</code> algorithm as this is what my keys have been generated with, change that to your will if that is not what you have been using. I like me your not sure what algorithm you used, it's been two years since I have done, you can check the content of you private key, in Bind9 format, the second line indicates the algorithm.</p>
<div class="highlight"><pre><span></span>$ cat /var/lib/knot/zones/bksp.space.d/Kbksp.space.+007+05001.private
Private-key-format: v1.3
Algorithm: <span class="m">7</span> <span class="o">(</span>NSEC3RSASHA1<span class="o">)</span>
....
</pre></div>
<p>The next configuration items is related to the signature and keys lifetime, I am neither a security nor DNS expert but rotating the signatures every week and the zone signing key every month looked like sane values. Next I enabled NSEC3 records to prevent <a href="https://appsecco.com/books/subdomain-enumeration/active_techniques/zone_walking.html">zone walking</a>. Finally I temporarily set the key management to manual as I will manually import the keys later and I wanted to test the zone as early as possible.</p>
<div class="highlight"><pre><span></span><span class="nt">policy</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">default</span>
<span class="nt">algorithm</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">rsasha1-nsec3-sha1</span>
<span class="nt">rrsig-lifetime</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">14d</span>
<span class="nt">rrsig-refresh</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">7d</span>
<span class="nt">zsk-lifetime</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">30d</span>
<span class="nt">dnskey-ttl</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">24h</span>
<span class="nt">nsec3</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">on</span>
<span class="nt">nsec3-iterations</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">10</span>
<span class="nt">nsec3-opt-out</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">on</span>
<span class="nt">nsec3-salt-lifetime</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">14d</span>
<span class="nt">nsec3-salt-length</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">40</span>
<span class="nt">manual</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">on</span>
</pre></div>
<p>A nice feature about Knot DNS is the <a href="https://www.knot-dns.cz/docs/3.0/html/configuration.html#zone-templates"><em>configuration templates</em></a> that allows to define template of configuration that you will apply to zones to quickly configure zones with similar configurations.</p>
<p>I defined two templates for primary, or master, zones, <code>default</code> and <code>signed</code> the only difference between the two begin the last to additional lines of the <code>signed</code> template. The first enable automatic DNSSEC signing and the second one refer to the DNSSEC policy previously created.</p>
<p>Now for the remaining fields, I indicate again the storage directory and then the zone file path, <code>%s</code> will be replaced by the zone name. <a href="https://www.knot-dns.cz/docs/3.0/html/operation.html#journal-behaviour">The journal</a> here refers to an database backed changelog to persist changes between restarts if they have not been flushed to the zone file. Here the value is <code>all</code> and the automatic flush to file is disabled by setting a value of -1 to <code>zonefile-sync</code> period. Means that the whole zone is stored in the journal and the original zone file is never altered by dynamic update like zone signing (or <a href="https://en.wikipedia.org/wiki/Dynamic_DNS">DDNS</a> but I am not currently using that). As stated in <a href="https://www.knot-dns.cz/docs/3.0/html/operation.html#example-1">the documentation</a>:</p>
<blockquote>
<p>Some users dislike that the server overwrites their prettily prepared zone file.</p>
</blockquote>
<p>I am one of those users.</p>
<p>Finally <code>difference-no-serial</code> means that the serial is automatically updated when reloading the zone file content using the <code>dateserial</code> format (<code>YYYYMMDDNN</code>).</p>
<p>I haved also prepared a <code>secondary</code> template for secondary or slave zone. In that case no file is used and the content of the zone is exclusively stored in the journal.</p>
<div class="highlight"><pre><span></span><span class="nt">template</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">default</span>
<span class="nt">storage</span><span class="p">:</span> <span class="s">"/var/lib/knot"</span>
<span class="nt">file</span><span class="p">:</span> <span class="s">"zones/%s.d/%s.zone"</span>
<span class="nt">journal-content</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">all</span>
<span class="nt">zonefile-load</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">difference-no-serial</span>
<span class="nt">zonefile-sync</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">-1</span>
<span class="nt">serial-policy</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">dateserial</span>
<span class="p p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">signed</span>
<span class="nt">storage</span><span class="p">:</span> <span class="s">"/var/lib/knot"</span>
<span class="nt">file</span><span class="p">:</span> <span class="s">"zones/%s.d/%s.zone"</span>
<span class="nt">journal-content</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">all</span>
<span class="nt">zonefile-load</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">difference-no-serial</span>
<span class="nt">zonefile-sync</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">-1</span>
<span class="nt">serial-policy</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">dateserial</span>
<span class="nt">dnssec-signing</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">on</span>
<span class="nt">dnssec-policy</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">default</span>
<span class="p p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">secondary</span>
<span class="nt">journal-content</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">all</span>
<span class="nt">zonefile-load</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">none</span>
<span class="nt">zonefile-sync</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">-1</span>
</pre></div>
<p>It is almost time to define the zone but I still need to do one thing before, defining remotes and ACLs to allow transfer to secondary name servers.</p>
<p>It might looks a bit confusing but here I have two dns servers acting both as primary and secondary servers for each other. So the remotes define the servers and the ACL the permission, I am using IP based ACL, I could of course be using <a href="https://www.knot-dns.cz/docs/3.0/html/reference.html#key-section">shared key</a> based ACL.</p>
<p>So I authorized the transfer of the zone to the ACL <code>bksp_secondary_acl</code> and authorized the zone to be transfer to me for the ACL <code>primary_acl</code>.</p>
<div class="highlight"><pre><span></span><span class="nt">remote</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">bksp_secondary</span>
<span class="nt">address</span><span class="p">:</span> <span class="p p-Indicator">[</span> <span class="nv">198.51.100.2</span><span class="p p-Indicator">,</span> <span class="nv">2001</span><span class="p p-Indicator">:</span><span class="nv">db8</span><span class="p p-Indicator">:</span><span class="nv">cafe</span><span class="p p-Indicator">:</span><span class="nv">bc68</span><span class="p p-Indicator">::</span><span class="nv">2</span> <span class="p p-Indicator">]</span>
<span class="p p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">primary</span>
<span class="nt">address</span><span class="p">:</span> <span class="p p-Indicator">[</span> <span class="nv">198.51.100.2</span><span class="p p-Indicator">,</span> <span class="nv">2001</span><span class="p p-Indicator">:</span><span class="nv">db8</span><span class="p p-Indicator">:</span><span class="nv">cafe</span><span class="p p-Indicator">:</span><span class="nv">bc68</span><span class="p p-Indicator">::</span><span class="nv">2</span> <span class="p p-Indicator">]</span>
<span class="nt">acl</span><span class="p">:</span>
<span class="p p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">bksp_secondary_acl</span>
<span class="nt">address</span><span class="p">:</span> <span class="p p-Indicator">[</span> <span class="nv">198.51.100.2</span><span class="p p-Indicator">,</span> <span class="nv">2001</span><span class="p p-Indicator">:</span><span class="nv">db8</span><span class="p p-Indicator">:</span><span class="nv">cafe</span><span class="p p-Indicator">:</span><span class="nv">bc68</span><span class="p p-Indicator">::</span><span class="nv">2</span> <span class="p p-Indicator">]</span>
<span class="nt">action</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">transfer</span>
<span class="p p-Indicator">-</span> <span class="nt">id</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">primary_acl</span>
<span class="nt">address</span><span class="p">:</span> <span class="p p-Indicator">[</span> <span class="nv">198.51.100.2</span><span class="p p-Indicator">,</span> <span class="nv">2001</span><span class="p p-Indicator">:</span><span class="nv">db8</span><span class="p p-Indicator">:</span><span class="nv">cafe</span><span class="p p-Indicator">:</span><span class="nv">bc68</span><span class="p p-Indicator">::</span><span class="nv">2</span> <span class="p p-Indicator">]</span>
<span class="nt">action</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">notify</span>
</pre></div>
<p>Then the zone configuration block looks like this, using the defined templates, remotes and ACLs.</p>
<div class="highlight"><pre><span></span><span class="nt">zone</span><span class="p">:</span>
<span class="c1"># Primary zones</span>
<span class="p p-Indicator">-</span> <span class="nt">domain</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">bksp.space</span>
<span class="nt">notify</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">bksp_secondary</span>
<span class="nt">acl</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">bksp_secondary_acl</span>
<span class="nt">template</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">signed</span>
<span class="c1"># Secondary zones</span>
<span class="p p-Indicator">-</span> <span class="nt">domain</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">example.com</span>
<span class="nt">master</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">primary</span>
<span class="nt">acl</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">primary_acl</span>
<span class="nt">template</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">secondary</span>
</pre></div>
<p>You can check you configuration using the <a href="https://www.knot-dns.cz/docs/3.0/html/man_knotc.html"><code>knotc</code></a> tool:</p>
<div class="highlight"><pre><span></span>$ knotc conf-check
Configuration is valid
</pre></div>
<h2>Importing the keys</h2>
<p>The tool to manage keys is <a href="https://www.knot-dns.cz/docs/3.0/html/man_keymgr.html"><code>keymgr</code></a>, to import the key run <code>keymgr <zone> import-bind</code> on both the KSK and the ZSK. Be sure that the files are owned by the <code>knot</code> user, if not it might not work. By default once imported the keys are stored in pem format in the <code>/var/lib/knot/keys/keys/</code> directory.</p>
<div class="highlight"><pre><span></span>$ keymgr bksp.space import-bind /var/lib/knot/bksp.space.d/Kbksp.space.+007+05001
$ keymgr bksp.space import-bind /var/lib/knot/bksp.space.d/Kbksp.space.+007+15557
</pre></div>
<p>If you want to be sure you can list the keys owned by a zone with:</p>
<div class="highlight"><pre><span></span>$ keymgr bksp.space list
<long hash> <span class="nv">ksk</span><span class="o">=</span>yes <span class="nv">zsk</span><span class="o">=</span>no <span class="nv">tag</span><span class="o">=</span><span class="m">05001</span> <span class="nv">algorithm</span><span class="o">=</span><span class="m">7</span> <span class="nv">size</span><span class="o">=</span><span class="m">4096</span> public-only<span class="o">=</span>no pre-active<span class="o">=</span><span class="m">0</span> <span class="nv">publish</span><span class="o">=</span><span class="m">1564190151</span> <span class="nv">ready</span><span class="o">=</span><span class="m">0</span> <span class="nv">active</span><span class="o">=</span><span class="m">1564190151</span> retire-active<span class="o">=</span><span class="m">0</span> <span class="nv">retire</span><span class="o">=</span><span class="m">0</span> post-active<span class="o">=</span><span class="m">0</span> <span class="nv">revoke</span><span class="o">=</span><span class="m">0</span> <span class="nv">remove</span><span class="o">=</span><span class="m">0</span>
<long hash> <span class="nv">ksk</span><span class="o">=</span>no <span class="nv">zsk</span><span class="o">=</span>yes <span class="nv">tag</span><span class="o">=</span><span class="m">05528</span> <span class="nv">algorithm</span><span class="o">=</span><span class="m">7</span> <span class="nv">size</span><span class="o">=</span><span class="m">2048</span> public-only<span class="o">=</span>no pre-active<span class="o">=</span><span class="m">0</span> <span class="nv">publish</span><span class="o">=</span><span class="m">1610134471</span> <span class="nv">ready</span><span class="o">=</span><span class="m">0</span> <span class="nv">active</span><span class="o">=</span><span class="m">0</span> retire-active<span class="o">=</span><span class="m">0</span> <span class="nv">retire</span><span class="o">=</span><span class="m">0</span> post-active<span class="o">=</span><span class="m">0</span> <span class="nv">revoke</span><span class="o">=</span><span class="m">0</span> <span class="nv">remove</span><span class="o">=</span><span class="m">0</span>
<long hash> <span class="nv">ksk</span><span class="o">=</span>no <span class="nv">zsk</span><span class="o">=</span>yes <span class="nv">tag</span><span class="o">=</span><span class="m">15557</span> <span class="nv">algorithm</span><span class="o">=</span><span class="m">7</span> <span class="nv">size</span><span class="o">=</span><span class="m">2048</span> public-only<span class="o">=</span>no pre-active<span class="o">=</span><span class="m">0</span> <span class="nv">publish</span><span class="o">=</span><span class="m">1564190124</span> <span class="nv">ready</span><span class="o">=</span><span class="m">0</span> <span class="nv">active</span><span class="o">=</span><span class="m">1564190124</span> retire-active<span class="o">=</span><span class="m">0</span> <span class="nv">retire</span><span class="o">=</span><span class="m">0</span> post-active<span class="o">=</span><span class="m">0</span> <span class="nv">revoke</span><span class="o">=</span><span class="m">0</span> <span class="nv">remove</span><span class="o">=</span><span class="m">0</span>
</pre></div>
<p>Here I have three keys because I have rotated the ZSK.</p>
<p>You can now reload Knot to take you changes in account and don't forget to check for potential errors by checking the service status. I everythin went well you should see logs telling that you zone has been signed.</p>
<div class="highlight"><pre><span></span>$ knotc reload
Reloaded
$ systemctl status knotc
</pre></div>
<p>Finally you can turn off the <code>manual</code> configuration entry to let Knot rotate the KSZ for you.</p>
<h2>Next steps</h2>
<p>Now that the ZSK rollover has been automated the final step for me would be to rotate the KSK in a similar manner. This one is more complex as the new DS record needs to be pushed in the parent zone. Fortunately, according to <a href="https://www.knot-dns.cz/docs/3.0/html/operation.html#automatic-ksk-and-zsk-rollovers-example">documentation</a> the key submission event is logged in a way that allow automation.</p>
<blockquote>
<p>If systemd is available, the KSK submission event is logged into journald in a structured way. The intended use case is to trigger a user-created script.</p>
</blockquote>
<p>So now the *only thing* that needs to be done is to write a script that update the DS using my <a href="https://api.ovh.com/console/#/domain/%7BserviceName%7D/dsRecord#POST">registrar API</a>, but that will probably have to wait ;).</p>
<hr>
<h3>References</h3>
<ul>
<li><a href="https://www.knot-dns.cz/docs/3.0/html/reference.html">Knot configuration reference</a></li>
<li><a href="https://www.knot-dns.cz/docs/3.0/html/operation.html">Knot operation guide</a></li>
<li><a href="https://www.knot-dns.cz/docs/3.0/html/configuration.html">Knot configuration guide</a></li>
<li><a href="https://gitlab.nic.cz/knot/knot-dns/">Source code and issue tracker</a></li>
<li><a href="https://www.knot-dns.cz/docs/3.0/html/migration.html#knot-dns-for-bind-users">Knot DNS for Bind users</a></li>
</ul>Compiling OpenCV with Structure From Motion module Python bindings2020-03-01T00:00:00-05:002020-03-01T00:00:00-05:00BlackSpongetag:bksp.space,2020-03-01:/blog/en/2020-03-01-compiling-opencv-with-structure-from-motion-module-python-bindings.html<p>Let's compile OpenCV with (actually working) Python bindings for the SFM module!</p><p><img alt="Blog header: stock footage hacker with angry meme face facing computer with opencv and python" src="https://bksp.space/blog/images/opencv-sfm-python.jpg"></p>
<p>About two weeks ago I was writing a application doing some camera motion estimation and I wanted to use the SFM OpenCV module to resolve the camera parameters. Now you can imagine what have been my surprise when I found out that not only the SFM module does not come with default OpenCV build but you need some dirty hack to make the Python bindings available.</p>
<p>I have then decided to write that down in case it can be of some benefit for someone else. Without further ado let's dive in the compiling pipeline of OpenCV (actually not very much).</p>
<h2>Installing dependencies</h2>
<p>The first step to compile OpenCV is installing its dependencies, this is identical from a <a href="https://docs.opencv.org/trunk/d7/d9f/tutorial_linux_install.html">standard OpenCV build</a>.</p>
<div class="highlight"><pre><span></span># apt install build-essential
# apt install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
# apt install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
</pre></div>
<p>This install the build dependencies and some optional libraries for modules. Then comes the dependencies of the SFM module, once again <a href="https://gregorkovalcik.github.io/opencv_contrib/tutorial_sfm_installation.html">nothing too special</a></p>
<div class="highlight"><pre><span></span># apt install libeigen3-dev libgflags-dev libgoogle-glog-dev libatlas-base-dev libsuitesparse-dev
</pre></div>
<p>The only thing is that one library is not in the packages repository and have to be compiled, so here we go. First let's create a directory for all the cloned repos to be put on.</p>
<div class="highlight"><pre><span></span>$ mkdir ~/opencv_build
$ cd ~/opencv_build
</pre></div>
<p>And then clone and install Ceres. By the way, if you wonder, this lib is used to solve the non linear quadratic optimisation problem involve in find the camera parameters.</p>
<div class="highlight"><pre><span></span>$ git clone https://ceres-solver.googlesource.com/ceres-solver
$ cd ceres-solver
$ mkdir build && cd build
$ make -j4
$ sudo make install
</pre></div>
<p>If you have a CPU with more than four threads then change the <code>4</code> in the <code>make -j4</code> above accordingly (just so you can use your full power).
Now we are supposed to be all good, if not <code>cmake</code> will tell you anyway.</p>
<h2>Patching SFM</h2>
<p>Let's clone the <code>opencv</code> and <code>opencv_contrib</code> module, if you prefer to use a more stable version than the <code>master</code>, you can download a <a href="http://opencv.org/releases.html">stable version</a> and extract it in your working directory instead.</p>
<div class="highlight"><pre><span></span>$ cd ~/opencv_build
$ git clone https://github.com/opencv/opencv.git
$ git clone https://github.com/opencv/opencv_contrib.git
</pre></div>
<p>Then edit with your favorite text editor the <code>reconstruct.hpp</code> header file.</p>
<div class="highlight"><pre><span></span>vim opencv_contrib/modules/sfm/include/opencv2/sfm/reconstruct.hpp
</pre></div>
<p>Now with the upmost care (because I like dramatic effect) change the <code>CV_EXPORTS</code> for something more sensible because right now the bindings are not being exported.
If you are gonna use only one of the overloaded <code>reconstruct</code> you could safely replace the one of your needs by <code>CV_EXPORTS_W</code> but if, like me, you want several bindings to be available then you need to rename the function using <code>CV_EXPORTS_AS</code>. That being said you will loose compatibility with other version of OpenCV depending of the way it has been compiled.</p>
<p>So in my case what I have done is something like this.</p>
<div class="highlight"><pre><span></span><span class="c1">// Fist declaration</span>
<span class="n">CV_EXPORTS_AS</span><span class="p">(</span><span class="n">reconstructProj</span><span class="p">)</span>
<span class="kt">void</span>
<span class="n">reconstruct</span><span class="p">(...);</span>
<span class="c1">// Second declaration</span>
<span class="n">CV_EXPORTS_AS</span><span class="p">(</span><span class="n">reconstructRotTrans</span><span class="p">)</span>
<span class="kt">void</span>
<span class="n">reconstruct</span><span class="p">(...);</span>
<span class="c1">// Nothing changed for the third and fourth.</span>
</pre></div>
<p>Note that I am exporting only the first two declarations as it is what I need but you could totally export the remaining two. Also the name given to <code>CV_EXPORTS_AS</code> is the name that we will be using in Python, the C++ usage remains the same.</p>
<p>The functions parameters also need now to be changed. If untouched we will run in some dimension problems as some output arrays (<code>Ps</code>, <code>Rs</code>, <code>Ts</code>, <code>points3d</code>) are actually arrays of arrays, the three first are arrays of transformation matrices and the fourth is an array of vectors.</p>
<p>Finally I have removed one of the <code>ifdef</code> precondition statements to force the declaration of the functions, not seen at the Python binding compilation stage otherwise. In the end my final <code>reconstruct.hpp</code> file looks like that (without documentation comments). Note the <code>OutputArrayOfArrays</code> in the declaration!</p>
<p>Again you can apply the change to the last two declaration to fit your needs.</p>
<div class="highlight"><pre><span></span><span class="cp">#ifndef __OPENCV_SFM_RECONSTRUCT_HPP__</span>
<span class="cp">#define __OPENCV_SFM_RECONSTRUCT_HPP__</span>
<span class="cp">#include</span> <span class="cpf"><vector></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><string></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><opencv2/core.hpp></span><span class="cp"></span>
<span class="k">namespace</span> <span class="n">cv</span>
<span class="p">{</span>
<span class="k">namespace</span> <span class="n">sfm</span>
<span class="p">{</span>
<span class="c1">//! @addtogroup reconstruction</span>
<span class="c1">//! @{</span>
<span class="hll"><span class="n">CV_EXPORTS_AS</span><span class="p">(</span><span class="n">reconstructProj</span><span class="p">)</span>
</span><span class="kt">void</span>
<span class="hll"><span class="n">reconstruct</span><span class="p">(</span><span class="n">InputArrayOfArrays</span> <span class="n">points2d</span><span class="p">,</span> <span class="n">OutputArrayOfArrays</span> <span class="n">Ps</span><span class="p">,</span> <span class="n">OutputArrayOfArrays</span> <span class="n">points3d</span><span class="p">,</span>
</span> <span class="n">InputOutputArray</span> <span class="n">K</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">is_projective</span> <span class="o">=</span> <span class="nb">false</span><span class="p">);</span>
<span class="hll"><span class="n">CV_EXPORTS_AS</span><span class="p">(</span><span class="n">reconstructRotTrans</span><span class="p">)</span>
</span><span class="kt">void</span>
<span class="hll"><span class="n">reconstruct</span><span class="p">(</span><span class="n">InputArrayOfArrays</span> <span class="n">points2d</span><span class="p">,</span> <span class="n">OutputArrayOfArrays</span> <span class="n">Rs</span><span class="p">,</span> <span class="n">OutputArrayOfArrays</span> <span class="n">Ts</span><span class="p">,</span>
</span><span class="hll"> <span class="n">InputOutputArray</span> <span class="n">K</span><span class="p">,</span> <span class="n">OutputArrayOfArrays</span> <span class="n">points3d</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">is_projective</span> <span class="o">=</span> <span class="nb">false</span><span class="p">);</span>
</span>
<span class="n">CV_EXPORTS</span>
<span class="kt">void</span>
<span class="nf">reconstruct</span><span class="p">(</span><span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">images</span><span class="p">,</span> <span class="n">OutputArray</span> <span class="n">Ps</span><span class="p">,</span> <span class="n">OutputArray</span> <span class="n">points3d</span><span class="p">,</span>
<span class="n">InputOutputArray</span> <span class="n">K</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">is_projective</span> <span class="o">=</span> <span class="nb">false</span><span class="p">);</span>
<span class="n">CV_EXPORTS</span>
<span class="kt">void</span>
<span class="nf">reconstruct</span><span class="p">(</span><span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">images</span><span class="p">,</span> <span class="n">OutputArray</span> <span class="n">Rs</span><span class="p">,</span> <span class="n">OutputArray</span> <span class="n">Ts</span><span class="p">,</span>
<span class="n">InputOutputArray</span> <span class="n">K</span><span class="p">,</span> <span class="n">OutputArray</span> <span class="n">points3d</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">is_projective</span> <span class="o">=</span> <span class="nb">false</span><span class="p">);</span>
<span class="c1">//! @} sfm</span>
<span class="p">}</span> <span class="cm">/* namespace cv */</span>
<span class="p">}</span> <span class="cm">/* namespace sfm */</span>
<span class="cp">#endif</span>
</pre></div>
<h2>Compiling OpenCV</h2>
<p>Then we can resume our compilation work, change directory to the <code>opencv</code> directory and compile it.</p>
<div class="highlight"><pre><span></span>$ cd ~/opencv
$ mkdir build
$ cd build
$ cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local -D OPENCV_EXTRA_MODULES_PATH=~/opencv_build/opencv_contrib/modules -D BUILD_SHARED_LIBS=ON -D BUILD_opencv_sfm=ON -D OPENCV_ENABLE_NONFREE=ON -D BUILD_SHARED_LIBS=ON -D BUILD_EXAMPLES=OFF -D BUILD_TESTS=OFF -D OPENCV_GENERATE_PKGCONFIG=ON ..
$ make -j4
$ sudo make install
</pre></div>
<p>If no error happened during the build steps (and hopefully it is the case) you should be able to access the sfm module within Python!</p>
<div class="highlight"><pre><span></span>$ python
Python 3.8.1 (default, Jan 22 2020, 06:38:00)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.sfm.reconstructProj
<built-in function reconstructProj>
</pre></div>
<hr>
<h3>References</h3>
<ul>
<li><a href="https://docs.opencv.org/4.2.0/d7/d9f/tutorial_linux_install.html">OpenCV tutorial on OpenCV install</a></li>
<li><a href="https://docs.opencv.org/4.2.0/db/db8/tutorial_sfm_installation.html">OpenCV tutorial on sfm install</a></li>
<li><a href="https://docs.opencv.org/4.2.0/da/d49/tutorial_py_bindings_basics.html">OpenCV tutorial on Python bindings</a></li>
<li><a href="https://github.com/opencv/opencv_contrib/issues/1675">Issue #1675 on the opencv_contrib repo</a></li>
</ul>Installation de VM avec PXE et kickstart2018-09-21T00:00:00-04:002018-09-21T00:00:00-04:00BlackSpongetag:bksp.space,2018-09-21:/blog/fr/2018-09-21-installation-de-vm-avec-pxe-et-kickstart.html<p>Comment mettre en place l'installation automatique de machine virtuelle Centos en utilisant le démarrage sur le réseau avec PXE.</p><p><img alt="Blog header: BlackSponge presents « Fun with Virtual Machines - PXE Install »" src="https://bksp.space/blog/images/pxe-install.jpg"></p>
<h2>Comment que quoi ?</h2>
<p>Installation et config :</p>
<ul>
<li>DHCP : pour indiquer l'adresse du serveur tftp et le nom du fichier pxe</li>
<li>TFTP : pour servir un menu et le noyau de l'os a installer</li>
<li>FTP (ou HTTP / NFS) : pour servir les fichiers d'installation de l'os</li>
</ul>
<h2>Salissons nous les mains</h2>
<p>D'abord il faut créer une vm avec deux interfaces réseau :</p>
<ul>
<li>une pour avoir internet (parce que ca peut etre pratique) configurer par exemple en mode NAT</li>
<li>une pour etre sur le réseau qui servira a déployer les machines configurer par exemple en mode private sans DHCP (vu qu'il sera fait par la machine qui est le serveur d'installation)</li>
</ul>
<p>Ici on va dire que la premiere s'appelle ens3 et la deuxieme ens4, que le premier réseau, <em>default</em>, est configuré en NAT avec dhcp avec ses ip dans 192.168.0.0/24 et que le deuxième, <em>private</em>, est configuré en mode private sans le dhcp et que son subnet sera 192.168.1.0/24.</p>
<p>D'abord pour configurer la deuxième interface en ip statique on va éditer le fichier <code>/etc/sysconfig/network-scripts/ens4</code> pour que le contenu soit le suivant :</p>
<div class="highlight"><pre><span></span>TYPE=Ethernet # Type de l'interface, ici de l'ethernet, rien a changer
BOOTPROTO=static # On passe en ip statique si c'etait en DHCP
NAME=ens4
DEVICE=ens4 # Le nom de la carte réseau, pas de raison de le changer
ONBOOT=yes # On active l'interface pour qu'elle démarre en meme temps que la machine
IPADDR=192.168.1.1 # IP de la machine
NETMASK=255.255.255.0 # Masque de sous réseau pour avoir un /24
</pre></div>
<p>Puis il n'y a plus qu'a redémarrer le réseau pour appliquer la configuration \o/</p>
<div class="highlight"><pre><span></span># systemctl restart NetworkManager
</pre></div>
<h3>Le DHCP</h3>
<p>Puisqu'il faut bien commencer quelque part pourquoi pas le DHCP, ca va etre le plus rapide a faire normalement. Pour ca il faut d'abord installer les paquets pour ca.</p>
<div class="highlight"><pre><span></span># yum install dhcp
</pre></div>
<p>Puis on va aller éditer le fichier de conf qui est <code>/etc/dhcp/dhcpd.conf</code>. Pour le moment on a pas besoin de faire quelque chose de bien complexe, seulement donner une ip aux VM, on verra le reste plus tard au besoin.</p>
<div class="highlight"><pre><span></span>subnet 192.168.1.0 netmask 255.255.255.0 { # On définit un réseau sur lequel le dhcp va assigner les IP
range 192.168.1.10 192.168.1.254; # Les IP vont de 192.168.1.10 à 192.168.1.254 (j'aime bien laisser quelques IP statiques dispo au cas ou)
}
</pre></div>
<p>Et voila ! Il faut maintenant activer le dhcp (pour qu'il démarre au boot) et le démarrer.</p>
<div class="highlight"><pre><span></span># systemctl enable dhcpd
# systemctl start dhcpd
</pre></div>
<p>Par la même occasion on va l'autoriser a passer le firewall puis on reload les règles du pare-feu pour qu'elles soient effectives.</p>
<div class="highlight"><pre><span></span># firewall-cmd --add-service=dhcp --permanent
# firewall-cml --reload
</pre></div>
<h3>Le serveur TFTP</h3>
<p>Maintenant le vrai fun commence et comme juste avant il faut commencer par installer le serveur tftp, on va aussi avoir besoin d'un autre paquet, syslinux, qui contient et programmes et noyaux pour pouvoir charger le boot en pxe et faire de jolis menus.</p>
<div class="highlight"><pre><span></span># yum install tftp-server syslinux
</pre></div>
<p>Le dossier depuis lequel les fichiers vont etre servis est <code>/var/lib/tftpboot</code>. Pour garder les choses bien ranger on va créer un dossier <code>pxelinux</code> dans ce dossier dans lequel on va copier le <code>pxelinux.0</code> de syslinux ainsi que le fichier <code>vesamenu.c32</code></p>
<div class="highlight"><pre><span></span># mkdir /var/lib/tftpboot/pxelinux
# cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/pxelinux/
# cp /usr/share/syslinux/vesamenu.c32 /var/lib/tftpboot/pxelinux/
</pre></div>
<p>Ensuite on va créer la configuration PXE. Pour ca on va créer un dossier <code>pxelinux.cfg</code> toujours dans le dossier <code>pxelinux</code>.</p>
<div class="highlight"><pre><span></span># mkdir /var/lib/tftpboot/pxelinux/pxelinux.cfg
</pre></div>
<p>Dans ce dossier on crée un fichier <code>default</code> avec le contenu suivant :</p>
<div class="highlight"><pre><span></span>default vesamenu.c32
prompt 0
timeout 600
label linux
menu label ^Install system
menu default
kernel ../images/centos7/vmlinuz
append initrd=../images/centos7/initrd.img ip=dhcp inst.repo=ftp://192.168.1.1/centos7/
</pre></div>
<p>Ce fichier sera le fichier par défaut utilisé par PXE, bien que ici ce ne soit pas nécessaire on pourrait faire une configuration par machine en utilisant leur MAC ou IP.</p>
<p>Aussi on vient d'anticiper légèrement sur la suite en spcéfiant le chemin vers le noyau a charger et l'url de l'endroit ou il y a les fichier d'installation.</p>
<p>On peut maintenant activer et démarrer le serveur TFTP comme on a fait pour le dhcp.</p>
<div class="highlight"><pre><span></span># systemctl enable tftp
# systemctl start tftp
</pre></div>
<p>Avant de passer tout de suite a la suite on va revenir un peu sur le dhcp pour rajouter les lignes spécique au PXE.</p>
<div class="highlight"><pre><span></span>subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.10 192.168.1.254;
next-server 192.168.1.1; # Adresse du serveur TFTP
filename "pxelinux/pxelinux.0"; # Fichier à charger
}
</pre></div>
<p>Puis on redémarre le serveur DHCP.</p>
<div class="highlight"><pre><span></span># systemctl restart dhcpd
</pre></div>
<p>On peut aussi rajouter les regles firewall pour laisser passer le serveur tftp.</p>
<div class="highlight"><pre><span></span># firewall-cmd --add-service=tftp --permanent
# firewall-cmd --reload
</pre></div>
<p>Si on test maintenant de créer une vm sur le réseau <em>private</em> en selectionnant le Démarrage sur le réseau à la création on devrait pouvoir voir au bout de quelque seconde un écran comme le suivant (qui boucle parce que la machine ne peut pas encore trouver le kernel qu'on lui a dit).</p>
<p><img alt="menu de boot pxe" src="https://bksp.space/blog/images/pxe-install-menu.png"></p>
<h3>Servir les fichiers d'installation avec FTP (et le noyau en TFTP)</h3>
<p>Avant de commencer il nous faut récupérer les fichiers d'installation de l'os (ceux qui sont dans l'iso en fait). Le plus simple dans ce cas je trouve c'est de monter l'iso en tant que cdrom sur la machine virtuelle puis de les copier. Mais il est tout a fait envisageable de faire autre chose.</p>
<p>Pour ca on peut aller dans virt-manager et dans la vue détaillé de la machine virtuelle, à la rubrique <code>IDE CDROM</code> selectionner l'iso d'installation dans le champ <code>Source path</code>. Si il n'y a pas de cdrom disponible on peut en rajouter en cliquant sur <code>Add hardware</code> puis <code>Storage</code> et selectionner <code>CDROM</code> dans le champ <code>Device type</code>, il faudra surement eteindre la vm puis la rallumer pour voir la modification appliquée.</p>
<p>Aussi il est conseillé pour eviter certains problèmes de permission d'avoir l'iso dans <code>/var/lib/libvirt/images</code> (au même endroit que les disques).</p>
<p>Une fois que c'est bon on peut monter le cdrom.</p>
<div class="highlight"><pre><span></span># mount /dev/cdrom /mnt
</pre></div>
<p>On crée maintenant le dossier qui va accueillir le noyau.</p>
<div class="highlight"><pre><span></span># mkdir -p /var/lib/tftpboot/images/centos7
</pre></div>
<p>Puis on copie les fichiers dedans.</p>
<div class="highlight"><pre><span></span># cp /mnt/images/pxeboot/{initrd.img,vmlinuz} /var/lib/tftpboot/images/centos7/
</pre></div>
<p>Maintenant que ça c'est fait on peut vraiment passer au serveur FTP avec comme d'habitude l'installation du paquet.</p>
<div class="highlight"><pre><span></span># yum install vsftpd
</pre></div>
<p>Ensuite on crée un dossier pour eviter de tout mettre en vrac puis on copie les fichiers.</p>
<div class="highlight"><pre><span></span># mkdir /var/ftp/centos7
# cp -r /mnt/* /var/ftp/centos7/
</pre></div>
<p>On active démarre le service et met les règles firewall.</p>
<div class="highlight"><pre><span></span># systemctl enable vsftpd
# systemctl start vsftpd
# firewall-cmd --add-service=ftp --permanent
# firewall-cmd --reload
</pre></div>
<p>A ce moment on devrait pouvoir faire démarer une machine sur l'installation de l'os. La vm peut avoir un peut de mal avec 1GiB de ram vu qu'elle doit charger pas mal de chose en mémoire, ça marche bien avec 2 GiB.</p>
<h2>Installation automatisée avec kickstart</h2>
<p>Dans le dossier des fichiers ftp, on va créer un dossier <code>ks</code>.</p>
<div class="highlight"><pre><span></span># mkdir /var/ftp/ks
</pre></div>
<p>Dans lequel on va placer un fichier <code>kickstart.cfg</code> avec le contenu suivant.</p>
<div class="highlight"><pre><span></span>#version=DEVEL
cmdline
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda
# Keyboard layouts
keyboard --vckeymap=fr --xlayouts='fr'
# System language
lang en_US.UTF-8
# Network information
network --bootproto=dhcp --device=eth0 --noipv6 --activate
network --hostname=guest
# Root password
rootpw --plaintext toto
# System services
services --disabled="chronyd"
# System timezone
timezone Europe/Paris --isUtc --nontp
# System bootloader configuration
bootloader --append=" crashkernel=auto" --location=mbr --boot-drive=vda
# Partition clearing information
clearpart --none --initlabel
# Disk partitioning information
part /boot --fstype="ext4" --ondisk=vda --size=1024
part swap --fstype="swap" --ondisk=vda --size=1024
part / --fstype="ext4" --ondisk=vda --size=8191
%packages
@^minimal
@core
kexec-tools
%end
%addon com_redhat_kdump --enable --reserve-mb='auto'
%end
</pre></div>
<p>Il faut noter que certaines option peuvent changer en fonction de la configuration de la vm :</p>
<ul>
<li>nom du disque d'installation : vda pour un os type Centos, hda pour du generic</li>
<li>nom de la carte réseau : eth0 pour une carte type virtio et ens3 pour rtl8139</li>
<li>la taille du disque d'installation (le partitionnage est configuré dans cet conf pour un disque de 10GiB)</li>
</ul>
<p>Pour plus d'info sur les commandes dans le kickstart : <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/installation_guide/sect-kickstart-syntax">https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/installation_guide/sect-kickstart-syntax</a></p>
<p>Pour vérifier les erreurs de syntaxes sans avoir a installer des millions de vm on utiliser la commande <code>ksvalidator</code> du package <code>pykickstart</code>.</p>
<div class="highlight"><pre><span></span># yum install pykickstart
# ksvalidator /var/ftp/ks/kickstart.cfg
</pre></div>
<p>Puis on peut editer la configuration PXE pour dire d'utiliser le fichier kickstart qu'on vient de créer en rajoutant <code>inst.ks=ftp://192.168.1.1/ks/kickstart.cfg</code> a la fin de la ligne qui commence par <code>append ...</code>.</p>
<div class="highlight"><pre><span></span>default vesamenu.c32
prompt 0
timeout 600
label linux
menu label ^Install system
menu default
kernel ../images/centos7/vmlinuz
append initrd=../images/centos7/initrd.img ip=dhcp inst.repo=ftp://192.168.1.1/centos7/ inst.ks=ftp://192.168.1.1/ks/kickstart.cfg
</pre></div>
<p>Et à partir de la si tout c'est bien passé on peut installer des vm automatiquement ! \o/</p>
<p><img alt="Victoire !" src="https://bksp.space/blog/images/victoire.gif"></p>