Opengrok, the source browser under Ubuntu, builds and integrates LDAP authentication login

-Foreword

Many small partners in android development know that opengrok is a sharp tool for fast code retrieval. It is very convenient to retrieve code. I have built opengrok many times before, but recently, due to the consideration of code security, I finally completed the ldap authentication of opengrok+tomcat, so I summarize and record it together.

There are two ways to install opengrok. One is docker installation, which is simple and fast, but it is not recommended for authentication and authorization, including large projects (in fact, the measured AOSP code is no problem, depending on your server performance). For reference: https://github.com/oracle/opengrok/tree/master/docker#readmehttps://github.com/oracle/opengrok/tree/master/docker#readme

Another installation method is the integration of ubuntu+opengrok+tomcat, which will be introduced in this example. Essentially, opengrok does not support account authentication and authorization. Therefore, this example uses Tomcat account authentication and opengrok to configure account authorization.

preparation

1. opengrok installation package: Releases · oracle/opengrok · GitHub   Our latest opengrok 1.7.18 next week   

2. In this example, the ubuntu version   Ubuntu 20.04.2 LTS

3. According to the official description, this version of opengrok requires JDK 11 or above. If you don't download the deb package directly on the official website, install it manually  

Java Downloads | Oracle 

Because oracle   ubuntu above JDK9 is no longer supported   Similar installation

sudo dpkg  -i ~/tools/jdk-11.0.11_linux-x64_bin.deb

4. Installation universal-ctags  , For specific installation reference, replace the default Exuberant of ubuntu   ctags Universal Ctags · GitHubUniversal Ctags has 13 repositories available. Follow their code on GitHub.https://github.com/universal-ctags

  5. Tools above Git 2.6 need to be installed

sudo apt-get install git

  Tomcat installation

In general, Ubuntu 20.4   use   sudo apt-get   install   tomcat, but it is found that tomcat 9.0 is installed, but the opengrok official website requires tomcat above 10.0. Therefore, tomcat can only be installed manually. Refer to: How to install and configure Apache Tomcat 10 on Ubuntu 20.04 system_ Linux ECS_ Yunwangniu railway station

1, Download tomcat 10.1.0-M5

Download the latest Apache Tomcat from the official website
Apache Tomcat® - Apache Tomcat 10 Software Downloads

wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.0-M5/bin/apache-tomcat-10.1.0-M5.tar.gz

After downloading, unzip the apache-tomcat-10.1.0-MD5.tar.gz file:

tar xvf apache-tomcat-10.1.0-MD5.tar.gz

Move extracted files to / usr:

sudo mv apache-tomcat-10.1.0-M5 /usr/share/apache-tomcat

Create apache tomcat user:

sudo useradd -M -d /usr/share/apache-tomcat tomcat

sudo chown -R tomcat /usr/share/apache-tomcat

2, Configuring Apache Tomcat 10

Allow access to the Apache Tomcat UI from a trusted network / IP.

1. Edit file:

sudo vim /usr/share/apache-tomcat/webapps/manager/META-INF/context.xml

sudo vim /usr/share/apache-tomcat/webapps/host-manager/META-INF/context.xml

Modify the allow row to add the IP from which you want to access the UI interface:

allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1|192.168.1.20" />

Alternatively, comment out the IP address restrictions to allow connections from anywhere:

<!--<Valve className="org.apache.catalina.valves.RemoteAddrValve"

         allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />-->

2. Secure access administrator / Dashboard

We need to protect access to the tomcat UI management area and edit the file:

sudo vim /usr/share/apache-tomcat/conf/tomcat-users.xml

Add the following before < / Tomcat users >:

<!-- manager section user role -->

<role rolename="manager-gui" />

<user username="manager" password="StronPassw0rd123" roles="manager-gui" />

<!-- admin section user role -->

<role rolename="admin-gui" />

<user username="admin" password="StronPassw0rd123" roles="manager-gui,admin-gui" />

3. Configure system services

To create a system D unit file for Tomcat 10:

sudo vim /etc/systemd/system/tomcat.service

Then paste the following into the file:

[Unit]
Description=Tomcat
After=syslog.target network.target
Documentation=https://tomcat.apache.org/tomcat-9.0-doc/index.html
RequiresMountsFor=/var/log/tomcat10 /var/log/tomcat10

[Service]
Type=forking
User=tomcat
Group=tomcat
Environment="JAVA_OPTS=-Djava.awt.headless=true"
Environment="CATALINA_TMPDIR=/tmp"
Environment=JDK_JAVA_OPTIONS="--add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-exports java.base/jdk.internal.misc=ALL-UNNAMED --add-exports java.base/sun.nio.ch=ALL-UNNAMED"
Environment="JAVA_HOME=/usr/lib/jvm/default-java"
Environment="JAVA_HOME=/usr/lib/jvm/jdk-11.0.11/"
Environment="CATALINA_HOME=/usr/share/apache-tomcat"
#Environment=CATALINA_BASE=/usr/share/apache-tomcat
Environment='CATALINA_BASE=/var/lib/tomcat10'
Environment='CATALINA_PID=/var/lib/tomcat10/tomcat.pid'
ExecStart=/usr/share/apache-tomcat/bin/catalina.sh start
ExecStop=/usr/share/apache-tomcat/bin/catalina.sh stop

# Logging
SyslogIdentifier=tomcat10

# Security
User=tomcat
Group=tomcat
PrivateTmp=true
NoNewPrivileges=yes
AmbientCapabilities=CAP_NET_BIND_SERVICE
CacheDirectory=tomcat10
CacheDirectoryMode=750
ProtectSystem=strict
#ReadWritePaths=/etc/tomcat10/Catalina/
ReadWritePaths=/var/lib/tomcat10/webapps/
ReadWritePaths=/var/log/tomcat10/
ReadWritePaths=/var/log/tomcat10/webapps/lib
ReadWritePaths=/home/jenkins/opengrok/
ReadWritePaths=/usr/share/apache-tomcat/conf/
ReadWritePaths=/home/jenkins/opengrok/index/suggester/


[Install]
WantedBy=multi-user.target

Reload the systemd daemon:

sudo systemctl daemon-reload

Run the following command to start the tomcat service:

sudo systemctl restart tomcat

Enable tomcat to start when the system boots:

sudo systemctl enable tomcat

If the startup is successful, you should see the 8080 port being used by the Java process on the system:

sudo ss -tunelp | grep 8080
[sudo] password for jenkins:
tcp    LISTEN  0       100                       *:8080                 *:*      users:(("java",pid=808616,fd=40)) uid:998 ino:45799734 sk:54 v6only:0 <->

Try to access port 8080 on your Web browser using the server IP address or its host name:

  opengrok installation

  1, Installation preparation

1. Create ~ / opengrok / to save the installation package, and configure the index file and source code

mkdir /opengrok/{src,data,dist,etc,log}

Download the installation package and unzip it to ~ / opengrok/dist/
Releases · oracle/opengrok · GitHubhttps://github.com/oracle/opengrok/releases

​wget /https://github.com/oracle/opengrok/releases/download/1.7.18/opengrok-1.7.18.tar.gz
tar -C /opengrok/dist --strip-components=1 -xzf opengrok-X.Y.Z.tar.gz

  2,   Project configuration (optional)

The default log configuration is copied from dist

cp ~/opengrok/dist/doc/logging.properties ~/opengrok/etc

The log configuration file is as follows:

handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler

java.util.logging.FileHandler.pattern = ~/opengrok/log/opengrok%g.%u.log
java.util.logging.FileHandler.append = false
java.util.logging.FileHandler.limit = 0
java.util.logging.FileHandler.count = 30
java.util.logging.FileHandler.level = ALL
java.util.logging.FileHandler.formatter = org.opengrok.indexer.logger.formatter.SimpleFileLogFormatter

java.util.logging.ConsoleHandler.level = WARNING
java.util.logging.ConsoleHandler.formatter = org.opengrok.indexer.logger.formatter.SimpleFileLogFormatter

org.opengrok.level = FINE

Add - Djava.util.logging.config.file during index synchronization   To enable log saving, refer to opengrok   index

opengrok   Project deployment

Now it's time to deploy opengrok as a tomcat web application. There are two ways:

1. Manually copy to / var/lib/tomcat10/webapps / directory, and Tomcat will be deployed automatically

sudo cp ~/opengrok/dist/lib/source.war  /var/lib/tomcat10/webapps/ 

Browse the web http://localhost:8080/source , you will be prompted that the configuration cannot be found. This needs an image index to be generated. The default directory will be / var/opengrok/etc/configuration.xml

To manually modify the configuration.xml path, you can modify the CONFIGURATION field in the / var/lib/tomcat10/webapps/source/WEB-INF/web.xml file, as follows:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee
         http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd"
         version="3.1">

    <display-name>OpenGrok</display-name>
    <description>A wicked fast source browser</description>
    <context-param>
        <description>Full path to the configuration file where OpenGrok can read its configuration</description>
        <param-name>CONFIGURATION</param-name>
        <param-value>/opengrok/etc/configuration.xml</param-value>
    </context-param>
...

  2. Script deployment

The actual effect is the same as manual copy, -c   CONFIGURATION can be specified

opengrok-deploy -c ~/opengrok/etc/configuration.xml \
    ~/opengrok/dist/lib/source.war /var/lib/tomcat10/webapps

opengrok project index

1. Create the required directories index and src

~/opengrok/index   And ~ / opengrok / index / adviser /, please ensure that opengrok   Have access to the above directories, otherwise the index will not succeed, or there are no candidates for search

mkdir -p ~/opengrok/src/ ~/opengrok/index/suggester

chmod 777 ~/opengrok/index/suggester 

Download two sample projects for testing

cd ~/opengrok/src

# use one of the training modules at GitHub as an example small app.      
git clone https://github.com/githubtraining/hellogitworld.git

# use OpenGrok as an example large app
git clone https://github.com/OpenGrok/OpenGrok

2. Execute the index command  

opengrok_base="/home/jenkins/opengrok/"
java \
        -Djava.util.logging.config.file=${opengrok_base}/etc/logging.properties \
        -jar ${opengrok_base}/dist/lib/opengrok.jar \
        -c ${ctags_root} \
        -R ${opengrok_base}/etc/readonly-configuration.xml \
        -s ${opengrok_base}/src \
        -d ${opengrok_base}/index -H -P -S -G --progress -v \
        -U "http://localhost:8080/source" \
        -W ${opengrok_base}/etc/configuration.xml

When indexing is complete, the browser   visit http://localhost:8080/source

3. Timing index

Timing synchronization using crontab command

* 0 * * * /home/jenkins/opengrok/opengrok_index.sh > /home/jenkins/opengrok/log/opengrok_index.lo

LDAP authentication configuration

My authentication strategy is relatively simple. The account of a specific group can only be accessed by login. As mentioned at the beginning, Tomcat realizes account authentication and opengrok realizes authorization.

1,Tomcat   Configure server.xml, ldap authentication needs to use JNDIRealm 

Open / var/lib/tomcat10/conf/server.xml and delete UserDatabase  

<!--

<Resource name="UserDatabase" auth="Container"

type="org.apache.catalina.UserDatabase"

    description="User database that can be updated and saved"

    factory="org.apache.catalina.users.MemoryUserDatabaseFactory"

    pathname="conf/tomcat-users.xml" />

-->

Delete UserDatabaseRealm in < Engine name = "Catalina" defaulthost = "localhost" > and add < realm classname = "org. Apache. Catalina. Realm. Jndireal". I add it to the Engine here. In fact, it can also be added to the Host and Context, but the influence range of permissions is different.

    <Engine name="Catalina" defaultHost="localhost">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

      <!-- Use the LockOutRealm to prevent attempts to guess user passwords
           via a brute-force attack -->
        <!-- This Realm uses the UserDatabase configured in the global JNDI
             resources under the key "UserDatabase".  Any edits
             that are performed against this UserDatabase are immediately
             available for use by the Realm.-

   <!-- comment for ldap
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>

         <Realm className="org.apache.catalina.realm.JNDIRealm" debug="99"
   connectionURL="ldap://ldap.china:389"
                                userBase="OU=Android Development Department,cn=Directory Manager"
                                userSearch="(sAMAccountName={0})"
                                userSubtree="true"
                                connectionName="username"
                                connectionPassword="Tpass"
                />-
  >-<Valve className="org.apache.catalina.valves.AccessLogValve"
 pattern="%h %l %u %t &quot;%r&quot; %s %b %{Authorization}i"
         prefix="catalina_access_log" suffix=".txt"/>

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

Parameter name

Realm   attribute

Parameter value

notes

LDAP   link   URLconnectionURLldap://localhost:389LDAP server and   Tomcat   On the same machine.389   by   Port of LDAP server

LDAP   Bind user

connectionName

cn=Directory Manager

Sun Directory Server   Directory administrator for         

Bind user password

connectionPassword

Secret

User search template

userPattern

uid={0},ou=People,dc=example,dc=com

User password

userPassword

userPassword

The password when the user logs in,   by   LDAP   Item in   userPassword   attribute

Role search path

roleBase

ou=Groups,dc= example,dc= com

 

Role properties

roleSearch

(uniqueMember={0})

Members in corresponding roles

Search role subdirectories

roleSubtree

False

Can not be set

 Apache Tomcat 10 (10.0.11) - Realm Configuration How-To

Ldapwiki: Tomcat And LDAP
Can refer to Using LDAP for user authentication _davidgjm in Tomcat 5.5 - CSDN blog

2. opengrok permission authorization configuration

insert.xml   The following insert.xml needs to be embedded into webapp/WEB-INF/web.xml of opengrok, which can be added manually or inserted with a command. Login config is used to set the login method of web page entry. If it takes effect, the user will be prompted to enter an account and password to access it

<?xml version="1.0" encoding="UTF-8"?>
<web-app>
   <!-- insert some elements here -->

<security-constraint>
    <web-resource-collection>                                               
        <web-resource-name>API endpoints are checked separately by the web app</web-resource-name>
        <url-pattern>/api/*</url-pattern>                                   
    </web-resource-collection>                                              
</security-constraint>

<security-constraint>
    <web-resource-collection>
        <web-resource-name>In general everything needs to be authenticated</web-resource-name>
        <url-pattern>/*</url-pattern> <!-- protect the whole application -->
        <url-pattern>/api/v1/search</url-pattern> <!-- protect search endpoint whitelisted above -->
        <url-pattern>/api/v1/suggest/*</url-pattern> <!-- protect suggest endpoint whitelisted above -->
    </web-resource-collection>

    <auth-constraint>
        <role-name>*</role-name>
    </auth-constraint>

    <user-data-constraint>
        <!-- transport-guarantee can be CONFIDENTIAL, INTEGRAL, or NONE -->
        <transport-guarantee>NONE</transport-guarantee>
    </user-data-constraint>
</security-constraint>

<security-role>
    <role-name>*</role-name>
</security-role>

<login-config>
    <auth-method>BASIC</auth-method>
</login-config>
</web-app>

  Insert with the following command:

 opengrok-deploy --insert insert.xml ~/opengrok/dist/lib/source.war \
        /var/lib/tomcat10/source/webapps/source.war 

Then browse the web page localhost:8080/source / and the following prompt appears

  If there is no prompt, you need to check the configuration of web.xml. We find that some garbled codes will appear when inserting with script. It is recommended to insert manually

3 add configuration using LDAP plug-in

Add readonly configuration file

~/opengrok/etc/readonly-configuration.xml

<?xml version="1.0" encoding="UTF-8"?>
<java version="1.8.0_172" class="java.beans.XMLDecoder">
 <object class="org.opengrok.indexer.configuration.Configuration" id="Configuration0">

  <void property="pluginStack">
        <!-- The setup will be inherited to all sub-stacks -->
        <void property="setup">
            <void method="put">
                <string>configuration</string>
                <string>/opengrok/auth/config/ldap-plugin-config-corp.xml</string>
            </void>
        </void>

        <!-- Place for authorization sub-stacks and/or authorization plugins -->

    </void>

   <!-- Authorization config end -->

 </object>
</java>

ldap-plugin-config-corp.xml

<?xml version="1.0" encoding="UTF-8"?>
<java version="1.8.0_65" class="java.beans.XMLDecoder">
        <object class="opengrok.auth.plugin.configuration.Configuration">
                <void property="interval">
                        <int>900000</int>
                </void>
                <void property="searchBase">
                        <string>dc=foo,dc=com</string>
                </void>
                <void property="webHooks">
                 <object class="opengrok.auth.plugin.util.WebHooks">
                  <void property="fail">
                   <object class="opengrok.auth.plugin.util.WebHook">
                    <void property="URI">
                     <string>http://localhost:8080/source/api/v1/messages</string>
                    </void>
                    <void property="content">                                                 
                     <string>{ "tags": [ "main" ], "cssClass": "class", "text": "corporate LDAP failed", "duration": "PT10M" }</string>
                    </void>
                   </object>
                  </void>
                 </object>
                </void>
                <void property="countLimit">
                 <int>10</int>
                </void>
                <void property="connectTimeout">
                 <int>3000</int>
                </void>
                <void property="searchTimeout">
                 <int>3000</int>
                </void>
                <void property="readTimeout">
                 <int>1000</int>
                </void>
                <void property="servers">
                        <void method="add">
                                <object class="opengrok.auth.plugin.ldap.LdapServer">
                                        <void property="name">
                                                <string>ldap://ldap.foo.com</string>
                                        </void>
                                        <void property="connectTimeout">
                                                <int>3000</int>
                                        </void>
                                </object>
                        </void>
                </void>
        </object>
</java>

The above configurations are configured through indexing, so you can refer to the index command   Index command, add.

Restart tomcat and browse localhost:8080/source to authenticate the ldap account.

Problem location:

The log of Tomcat is in / var/lib/tomcat10/log/

opengrok   Log ~ / opengrok/log

Tags: Android Tomcat

Posted on Tue, 28 Sep 2021 15:12:21 -0400 by Ph0enix