说明：

在StarRocts官网中，默认支持docker方式运行，受制于docker镜像下载限制。因此，本文章采取手动安装各组件，以实现StarRocts的数据湖运行。

一、环境信息

指标	参数
操作系统	Ubuntu server24
CPU	AMD Ryzen 7 3700U 4核
RAM	11GB
SSD	100GB
网卡	1 × NAT
环境	VMware Pro15
本机IP	192.168.179.131

信息项	值
用户名	vmware8x
密码	vmware8x
环境根目录	/home/vmware8x

组件	版本	链接
Hadoop	3.4.1-lean	下载地址
Hive	3.1.3	下载地址
StarRocks	3.4.1	下载地址
Iceberg	1.10.1 Spark 3.5_with Scala 2.12	下载地址

二、基本配置

1、更新源修改

$ sudo cp /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list.d/ubuntu.sources.bak
$ sudo vim /etc/apt/sources.list.d/ubuntu.sources
# 添加如下内容：
Types: deb
URIs: https://mirrors.tuna.tsinghua.edu.cn/ubuntu/
Suites: noble noble-updates noble-security
Components: main restricted universe multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

2、安装软件

$ sudo apt update
$ sudo apt upgrade 
$ sudo apt install tar
$ sudo apt install vim
$ sudo apt install gcc

3、系统配置

$ sudo vim /etc/sysctl.conf
vm.overcommit_memory=1
vm.swappiness=0
net.ipv4.tcp_abort_on_overflow=1
net.core.somaxconn=1024
vm.max_map_count = 262144
kernel.perf_event_paranoid=1

# 取消交换分区
$ sudo vim /etc/fstab
# /swap.img     none    swap    sw      0       0

$ sudo vim /etc/security/limits.conf
* soft nproc 65535
* hard nproc 65535
* soft nofile 655350
* hard nofile 655350
* soft stack unlimited
* hard stack unlimited
* hard memlock unlimited
* soft memlock unlimited

$ sudo apt-get install --reinstall tzdata
$ sudo dpkg-reconfigure tzdata
# 设置时区
Asia -> Shanghai

$ echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
$ echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

# IP配置
$ sudo vim /etc/netplan/50-cloud-init.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: no
      addresses:
        - 192.168.179.131/24
      routes:
        - to: 0.0.0.0/0
          via: 192.168.179.2
      nameservers:
        addresses:
          - 114.114.114.114

# 绑定IP与域名
$ sudo vim /etc/hosts
192.168.179.131 starRocks01
$ sudo vim /etc/hostname
starRocks01

# 必须重启系统否则数据湖功能异常
$ sudo reboot

三、组件安装与配置

1、JDK

$ tar -zxvf jdk-8u181-linux-x64.tar.gz
# 安装OpenJDK11，兼容StarRocts3.x
$ tar -zxvf openjdk-11.0.2_linux-x64_bin.tar.gz
$ sudo vim /etc/profile
export JAVA_HOME=/home/vmware8x/jdk-11.0.2
# export JAVA_HOME=/home/vmware8x/jdk1.8.0_181
export PATH=$PATH:$JAVA_HOME/bin
$ source /etc/profile

备注：JDK8可兼容Hadoop、Hive；但不兼容StarRocts3.x。OpenJDK11用于兼容StarRocts3.x。

2、MySQL

$ sudo apt install mysql-server -y
$ sudo systemctl start mysql
$ sudo systemctl enable mysql
$ sudo mysql_secure_installation
Securing the MySQL server deployment.

Connecting to MySQL using a blank password.

VALIDATE PASSWORD COMPONENT can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD component?

# 密码强度验证，y表示启用，其他键表示不启用
Press y|Y for Yes, any other key for No: n

Skipping password set for root as authentication with auth_socket is used by default.
If you would like to use password authentication instead, this can be done with the "ALTER_USER" command.
See https://dev.mysql.com/doc/refman/8.0/en/alter-user.html#alter-user-password-management for more information.

By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

# 删除匿名用户？（按y b| y为“是”，其他键为“否”）
Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.


Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

# 禁止root用户远程登录？（按y b| y为“是”，其他键为“否”）
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.

By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.

# 删除测试数据库并访问它？（按y b| y为“是”，其他键为“否”）
Remove test database and access to it? (Press y|Y for Yes, any other key for No) :  y
 - Dropping test database...
Success.

 - Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

# 之前的操作是否立即生效？（按y b| y为“是”，其他键为“否”）
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y


$ sudo mysql
mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'root';
mysql> FLUSH PRIVILEGES;
mysql> EXIT;

$ mysql -u root -p
# 此处需输入刚刚设置的密码
Enter password: 

3、Hadoop

$ tar -zxvf hadoop-3.4.1.tar.gz
$ mv hadoop-3.4.1 hadoop

$ sudo vim /etc/profile
# Hadoop
export HADOOP_HOME=/home/vmware8x/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CLASSPATH=`hadoop classpath`

$ source /etc/profile

$ hadoop version
Hadoop 3.4.1
Source code repository https://github.com/apache/hadoop.git -r 4d7825309348956336b8f06a08322b78422849b1
Compiled by mthakur on 2024-10-09T14:57Z
Compiled on platform linux-x86_64
Compiled with protoc 3.23.4
From source with checksum 7292fe9dba5e2e44e3a9f763fce3e680
This command was run using /home/vmware8x/hadoop/share/hadoop/common/hadoop-common-3.4.1.jar

$ vim /home/vmware8x/hadoop/etc/hadoop/core-site.xml
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://starRocks01:10001</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/vmware8x/data/hadoop</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.vmware8x.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.vmware8x.groups</name>
    <value>*</value>
  </property>
</configuration>

$ mkdir ~/data
$ mkdir ~/data/hadoop

$ vim /home/vmware8x/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

$ vim /home/vmware8x/hadoop/etc/hadoop/mapred-site.xml
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>

  <!- Configure MapReduce route -->
  <property>
    <name>yarn.app.mapreduce.am.env</name>
    <value>HADOOP_MAPRED_HOME=/home/vmware8x/hadoop</value>
  </property>
  <property>
    <name>mapreduce.map.env</name>
    <value>HADOOP_MAPRED_HOME=/home/vmware8x/hadoop</value>
  </property>
  <property>
    <name>mapreduce.reduce.env</name>
    <value>HADOOP_MAPRED_HOME=/home/vmware8x/hadoop</value>
  </property>
</configuration>

$ ​vim /home/vmware8x/hadoop/etc/hadoop/yarn-site.xml
<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>starRocks01</value>
  </property>
</configuration>

$ vim /home/vmware8x/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/home/vmware8x/jdk1.8.0_181
$ vim /home/vmware8x/hadoop/etc/hadoop/mapred-env.sh
export JAVA_HOME=/home/vmware8x/jdk1.8.0_181
$ vim /home/vmware8x/hadoop/etc/hadoop/yarn-env.sh
export JAVA_HOME=/home/vmware8x/jdk1.8.0_181

$ ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/authorized_keys
$ ssh localhost

$ hdfs namenode -format

$ start-dfs.sh
$ start-yarn.sh

$ jps
2754 DataNode
3155 ResourceManager
3462 Jps
3257 NodeManager
2649 NameNode
2927 SecondaryNameNode

启动完成后可通过如下地址访问

http://192.168.179.131:9870

访问Hadoop集群（伪分布）页面

http://192.168.179.131:8088/cluster

4、Hive

$ tar -zxvf apache-hive-3.1.3-bin.tar.gz
$ mv apache-hive-3.1.3-bin hive

$ sudo vim /etc/profile
# Hive
export HIVE_HOME=/home/vmware8x/hive
export PATH=$PATH:$HIVE_HOME/bin

$ source /etc/profile

$ cp /home/vmware8x/hive/conf/hive-env.sh.template /home/vmware8x/hive/conf/hive-env.sh

$ vim /home/vmware8x/hive/conf/hive-env.sh
export HADOOP_HOME=/home/vmware8x/hadoop
export HIVE_CONF_DIR=/home/vmware8x/hive/conf

$ touch /home/vmware8x/hive/conf/hive-site.xml
$ vim /home/vmware8x/hive/conf/hive-site.xml
<?xml version="1.0"?> 
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> 
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost/hive_metastore?createDatabaseIfNotExist=true&amp;useSSL=false&amp;allowPublicKeyRetrieval=true</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
  </property>

  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>root</value>
  </property>

  <property>
    <name>hive.security.authorization.enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>hive.users.in.admin.role</name>
    <value>vmware8x</value>
  </property>

</configuration> 

$ cp /home/vmware8x/hive/conf/hive-log4j2.properties.template /home/vmware8x/hive/conf/hive-log4j2.properties
$ vim hive-log4j.properties
property.hive.log.dir = /home/vmware8x/hive/logs

# 将mysql-connector-j-8.1.0.jar文件拷贝到/home/vmware8x/hive/lib中

$ schematool -initSchema -dbType mysql
$ nohup hive --service metastore &
$ nohup hive --service hiveserver2 &
# 测试是否连接成功
$ beeline
beeline> !connect jdbc:hive2://starRocks01:10000/default
beeline> SHOW DATABASES;
Connected to: Apache Hive (version 3.1.3)
Driver: Hive JDBC (version 3.1.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://starRocks01:10000/default> SHOW DATABASES;
INFO  : Compiling command(queryId=vmware8x_20250330105132_ee51973c-fb6e-498c-a7b3-f40de4c231ef): SHOW DATABASES
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling command(queryId=vmware8x_20250330105132_ee51973c-fb6e-498c-a7b3-f40de4c231ef); Time taken: 1.153 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=vmware8x_20250330105132_ee51973c-fb6e-498c-a7b3-f40de4c231ef): SHOW DATABASES
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=vmware8x_20250330105132_ee51973c-fb6e-498c-a7b3-f40de4c231ef); Time taken: 0.114 seconds
INFO  : OK
INFO  : Concurrency mode is disabled, not creating a lock manager
+----------------+
| database_name  |
+----------------+
| default        |
+----------------+
1 row selected (2.031 seconds)

5、StarRocts

# 请提前下载StarRocts3.4的TAR包，并上传到服务器或虚拟机指定位置
$ tar -zxvf ~/StarRocks-3.4.1-ubuntu-amd64.tar.gz
$ mv ~/StarRocks-3.4.1-ubuntu-amd64 ~/starRocks

$ mkdir /home/vmware8x/starRocks/fe/meta
$ mkdir /home/vmware8x/starRocks/be/storage

$ /home/vmware8x/starRocks/fe/conf/fe.conf
JAVA_HOME = /home/vmware8x/jdk-11.0.2
meta_dir = /home/vmware8x/starRocks/fe/meta
# priority_networks = 192.168.179.131/24
default_replication_num = 1

http_port = 10010
rpc_port = 10011
query_port = 10012
edit_log_port = 10013

$ /home/vmware8x/starRocks/be/conf/be.conf
JAVA_HOME = /home/vmware8x/jdk-11.0.2
storage_root_path = /home/vmware8x/starRocks/be/storage
priority_networks = 192.168.179.131/24
mem_limit = 80%
jit_lru_cache_size = 1073741824

be_port = 10014
be_http_port = 10015
heartbeat_service_port = 10016
brpc_port = 10017
starlet_port = 10018

$ sudo vim /etc/profile
# StarRocks
export STARROCKS_HOME=/home/vmware8x/starRocks
export PATH=$PATH:$STARROCKS_HOME/fe
export PATH=$PATH:$STARROCKS_HOME/be

$ source /etc/profile

$ start_fe.sh --daemon
验证是否启动成功：
①浏览器地址输入：http://<IP地址>:10010
②输入用户名：root（仅输入用户名，无需密码即可登录）

$ start_be.sh --daemon

$ mysql -h starRocks01 -u root -P 10012
mysql> show proc '/frontends'\G
*************************** 1. row ***************************
             Name: 192.168.179.131_10013_1743055655291
               IP: 192.168.179.131
      EditLogPort: 10013
         HttpPort: 10010
        QueryPort: 10012
          RpcPort: 10011
             Role: LEADER
        ClusterId: 1317244167
             Join: true
            Alive: true
ReplayedJournalId: 70
    LastHeartbeat: 2025-03-27 14:11:12
         IsHelper: true
           ErrMsg: 
        StartTime: 2025-03-27 14:07:44
          Version: 3.4.1-2f78e09
1 row in set (0.07 sec)

mysql> ALTER SYSTEM ADD BACKEND "starRocks01:10016";   # 一定要用大写，用小写时，BE和FE难以通信
mysql> SHOW PROC '/backends'\G
# 不需要就执行删除命令
mysql> ALTER SYSTEM DROP BACKEND "starRocks01:10016";   # 一定要用大写，用小写时，BE和FE难以通信

# 为登录MySQL的root用户设置密码，替换<password>（不含引号）为自定义的密码
# 此操作可跳过，因为此环境并非生产环境，安全需求极低
mysql> SET PASSWORD = PASSWORD('<password>');
# 推出MySQL后登录测试
mysq> exit;
# 如果选择了设置密码，则需要如下命令方可访问
# 请加入-p，此处是小写！！！
$ mysql -h starRocks01 -u root -P 10012 -p
Enter password: #此时需要输入密码

6、Spark

$ tar -zxvf ~/spark-3.5.5-bin-hadoop3.tgz
$ mv spark-3.5.5-bin-hadoop3 spark

$ sudo vim /etc/profile
# Spark
export SPARK_HOME=/home/vmware8x/spark
export PATH=$PATH:$SPARK_HOME/bin

$ source /etc/profile

$ cp /home/vmware8x/spark/conf/spark-env.sh.template /home/vmware8x/spark/conf/spark-env.sh
$ vim /home/vmware8x/spark/conf/spark-env.sh
export SPARK_DIST_CLASSPATH=$(/home/vmware8x/hadoop/bin/hadoop classpath)
export JAVA_HOME=/home/vmware8x/jdk-11.0.2

$ cp $HADOOP_HOME/etc/hadoop/core-site.xml $SPARK_HOME/conf/
$ cp $HADOOP_HOME/etc/hadoop/hdfs-site.xml $SPARK_HOME/conf/

# 将mysql-connector-j-8.1.0.jar文件拷贝到/home/vmware8x/spark/jars中
# 将iceberg-spark-runtime-3.5_2.12-1.8.1.jar放入到/home/vmware8x/spark/jars下
# 启动带有Iceberg的Spark
$ pyspark --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \
    --conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.demo.type=hive \
    --conf spark.sql.catalog.demo.uri=thrift://starRocks01:9083

四、服务启动

备注：通过第三部分的“组件安装与配置”的操作后，服务均被启动，无需再次启动。此部分的目的是建立一个启动列表，方便关机重启后的启动操作。

1、Hadoop服务

$ start-dfs.sh
$ start-yarn.sh

启动完成后可通过如下地址访问

http://192.168.179.131:9870

访问Hadoop集群（伪分布）页面

http://192.168.179.131:8088/cluster

2、StarRocks服务

$ start_fe.sh  --daemon

验证是否启动成功：
①浏览器地址输入：http://192.168.179.131:10010
②输入用户名：root（仅输入用户名，无需密码即可登录）

$ start_be.sh  --daemon

验证FE服务是否被启动

# 进入MySQL Client，密码为root
$ mysql -h 192.168.179.131 -u root -P 10012

# 退出MySQL服务
mysql> exit;

3、Hive服务

# 必须启动
$ nohup hive --service metastore &
# 必须启动
$ nohup hive --service hiveserver2 &
# 非必须启动
$ beeline
beeline> !connect jdbc:hive2://starRocks01:10000/default
Enter username for jdbc:hive2://starRocks01:10000/default: vmware8x
Enter password for jdbc:hive2://starRocks01:10000/default: ********
Connected to: Apache Hive (version 3.1.3)
Driver: Hive JDBC (version 3.1.3)
Transaction isolation: TRANSACTION_REPEATABLE_READ

4、Spark服务

# demo为该目录从 Hive 元存储中加载表
$ pyspark --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \
    --conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.demo.type=hive \
    --conf spark.sql.catalog.demo.uri=thrift://starRocks01:9083

# 端口检查命令，服务冲突时用于排查
$ ss -tuln | grep :<端口号>
$ ss -uln | grep :<端口号>