Compare commits
10 Commits
1b7a234f34
...
979c739118
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
979c739118 | ||
|
|
c1ce1c718c | ||
|
|
2a7a2a2976 | ||
|
|
f37ecd7211 | ||
|
|
52a1c98e41 | ||
|
|
d813d05f62 | ||
|
|
e2762bbb41 | ||
|
|
9be6d8606c | ||
|
|
abc7dab7cf | ||
|
|
98dca2ab64 |
15
README.md
15
README.md
@ -1,11 +1,24 @@
|
||||
# storm
|
||||
|
||||
#### 介绍
|
||||
A free and open source distributed realtime computation system
|
||||
Apache Storm是一个免费的开源分布式实时计算系统。 Storm可以轻松地处理无限数据流, 并且实时处理Hadoop批处理任务。 可以和任何编程语言一起使用,并且使用起来非常有趣!
|
||||
Storm有许多用例:实时分析,在线机器学习,连续计算,分布式RPC,ETL等等。 Storm是快速 的:一个基准计算每个节点每秒处理超过一百万个元组。 它具有可扩展性,容错性,可确保您的 数据得到处理,且易于设置和操作。
|
||||
Storm集成了您已经使用的队列和数据库技术。 Storm拓扑消耗数据流,并以任意复杂的方式处理 这些流,然后重新分配计算的每个阶段之间的流
|
||||
|
||||
#### 软件架构
|
||||
软件架构说明
|
||||
与Hadoop主从架构一样,Storm也采用Master/Slave体系结构,分布式计算由Nimbus和Supervisor两类服务进程实现,Nimbus进程运行在集群的主节点,负责任务的指派和分发,Supervisor运行在集群的从节点,负责执行任务的具体部分。
|
||||
- Nimbus: Storm集群的Master节点,负责资源分配和任务调度,负责分发用户代码,指派给具体的Supervisor节点上的Worker节点,去运行Topology对应的组件(Spout/Bolt)的Task。
|
||||
- Supervisor: Storm集群的从节点,负责接受Nimbus分配的任务,启动和停止属于自己管理的worker进程。通过Storm的配置文件中的supervisor.slots.ports配置项,可以指定在一个Supervisor上最大允许多少个Slot,每个Slot通过端口号来唯一标识,一个端口号对应一个Worker进程(如果该Worker进程被启动)。
|
||||
- Worker: 负责运行具体处理组件逻辑的进程。Worker运行的任务类型只有两种,一种是Spout任务,一种是Bolt任务。
|
||||
- Task: worker中每一个spout/bolt的线程称为一个task。同一个spout/bolt的task可能会共享一个物理线程,该线程称为executor。
|
||||
- ZooKeeper: 用来协调Nimbus和Supervisor,如果Supervisor因故障出现问题而无法运行Topology,Nimbus会第一时间感知到,并重新分配Topology到其它可用的Supervisor上运行
|
||||
|
||||
#### ARM支持:
|
||||
|
||||
1. 移植指南:xxxx
|
||||
2. 部署指南:https://support.huaweicloud.com/dpmg-apache-kunpengbds/kunpengstorm_04_0001.html
|
||||
3. 调优指南:https://support.huaweicloud.com/tngg-kunpengbds/kunpengstormhdp_05_0002.html
|
||||
|
||||
#### 安装教程
|
||||
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
BIN
apache-storm-1.2.4.tar.gzac
Normal file
BIN
apache-storm-1.2.4.tar.gzac
Normal file
Binary file not shown.
@ -3,15 +3,14 @@ Description=Apache Storm logviewer service
|
||||
After=syslog.target network.target
|
||||
|
||||
[Service]
|
||||
User=storm
|
||||
Group=storm
|
||||
EnvironmentFile=/etc/sysconfog/storm
|
||||
EnvironmentFile=/etc/sysconfig/storm
|
||||
ExecStart=/opt/storm/bin/storm --config /etc/storm/storm.yaml logviewer
|
||||
Restart=on-failure
|
||||
StandardOutput=syslog
|
||||
StandardError=syslog
|
||||
SyslogIdentifier=storm-logviewer
|
||||
SyslogLevel=debug
|
||||
SuccessExitStatus=143
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
@ -3,15 +3,15 @@ Description=Apache Storm nimbus service
|
||||
After=syslog.target network.target
|
||||
|
||||
[Service]
|
||||
User=storm
|
||||
Group=storm
|
||||
EnvironmentFile=/etc/sysconfog/storm
|
||||
EnvironmentFile=/etc/sysconfig/storm
|
||||
ExecStart=/opt/storm/bin/storm --config /etc/storm/storm.yaml nimbus
|
||||
Restart=on-failure
|
||||
StandardOutput=syslog
|
||||
StandardError=syslog
|
||||
SyslogIdentifier=storm-nimbus
|
||||
SyslogLevel=debug
|
||||
SuccessExitStatus=20
|
||||
SuccessExitStatus=143
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
@ -3,15 +3,15 @@ Description=Apache Storm supervisor service
|
||||
After=syslog.target network.target
|
||||
|
||||
[Service]
|
||||
User=storm
|
||||
Group=storm
|
||||
EnvironmentFile=/etc/sysconfog/storm
|
||||
EnvironmentFile=/etc/sysconfig/storm
|
||||
ExecStart=/opt/storm/bin/storm --config /etc/storm/storm.yaml supervisor
|
||||
Restart=on-failure
|
||||
StandardOutput=syslog
|
||||
StandardError=syslog
|
||||
SyslogIdentifier=storm-supervisor
|
||||
SyslogLevel=debug
|
||||
SuccessExitStatus=20
|
||||
SuccessExitStatus=143
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
@ -3,15 +3,14 @@ Description=Apache Storm ui service
|
||||
After=syslog.target network.target
|
||||
|
||||
[Service]
|
||||
User=storm
|
||||
Group=storm
|
||||
EnvironmentFile=/etc/sysconfog/storm
|
||||
EnvironmentFile=/etc/sysconfig/storm
|
||||
ExecStart=/opt/storm/bin/storm --config /etc/storm/storm.yaml ui
|
||||
Restart=on-failure
|
||||
StandardOutput=syslog
|
||||
StandardError=syslog
|
||||
SyslogIdentifier=storm-ui
|
||||
SyslogLevel=debug
|
||||
SuccessExitStatus=143
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
40
storm.spec
40
storm.spec
@ -4,7 +4,7 @@
|
||||
%define _log_dir %{_var}/log/storm
|
||||
%define _data_dir %{_sharedstatedir}/storm
|
||||
Name: storm
|
||||
Version: 1.2.3
|
||||
Version: 1.2.4
|
||||
Release: 2
|
||||
Summary: Apache Storm realtime computation system
|
||||
License: Apache-2.0
|
||||
@ -16,9 +16,9 @@ Source3: %{name}-logviewer.service
|
||||
Source4: %{name}-ui.service
|
||||
Source5: %{name}.logrotate
|
||||
Source6: %{name}.conf
|
||||
Source7: apache-storm-1.2.3.tar.gzaa
|
||||
Source8: apache-storm-1.2.3.tar.gzab
|
||||
Source9: apache-storm-1.2.3.tar.gzac
|
||||
Source7: apache-storm-1.2.4.tar.gzaa
|
||||
Source8: apache-storm-1.2.4.tar.gzab
|
||||
Source9: apache-storm-1.2.4.tar.gzac
|
||||
BuildRoot: %{_tmppath}/%{name}-%{version}-build
|
||||
%{?systemd_requires}
|
||||
BuildRequires: systemd
|
||||
@ -60,21 +60,15 @@ install -p -D -m 644 %{S:6} $RPM_BUILD_ROOT%{_sysconfdir}/sysconfig/storm
|
||||
install -p -D -m 644 conf/storm.yaml $RPM_BUILD_ROOT%{_conf_dir}/
|
||||
install -p -D -m 644 conf/storm_env.ini $RPM_BUILD_ROOT%{_conf_dir}/
|
||||
|
||||
%pre
|
||||
/usr/bin/getent group storm > /dev/null || /usr/sbin/groupadd -r storm
|
||||
if ! /usr/bin/getent passwd storm > /dev/null ; then
|
||||
/usr/sbin/useradd -r -g storm -m -d %{_prefix}/storm -s /bin/bash -c "Storm" storm
|
||||
fi
|
||||
|
||||
%post
|
||||
%systemd_post storm-nimbus.service
|
||||
%systemd_post storm-supervisor.service
|
||||
%systemd_post storm-logviewer.service
|
||||
%systemd_post storm-ui.service
|
||||
|
||||
%postun
|
||||
%preun
|
||||
for module in nimbus supervisor logviewer ui; do
|
||||
systemctl status storm.$module > /dev/null
|
||||
systemctl status storm-$module > /dev/null
|
||||
if [[ $? -eq 0 ]]; then
|
||||
%systemd_preun storm-$module.service
|
||||
fi
|
||||
@ -89,13 +83,25 @@ done
|
||||
%config(noreplace) %{_sysconfdir}/logrotate.d/storm
|
||||
%config(noreplace) %{_sysconfdir}/sysconfig/storm
|
||||
%config(noreplace) %{_conf_dir}/*
|
||||
%attr(-,storm,storm) %{_prefix}/storm
|
||||
%attr(0755,storm,storm) %dir %{_log_dir}
|
||||
%attr(0700,storm,storm) %dir %{_data_dir}
|
||||
%{_prefix}/storm
|
||||
%dir %{_log_dir}
|
||||
%dir %{_data_dir}
|
||||
|
||||
%changelog
|
||||
* Wed May 26 2021 wangchong <wangchong56@huawei.com> 1.2.3 - 2
|
||||
* Sun Jun 25 2023 xiexing <xiexing4@hisilicon.com> - 1.2.4-2
|
||||
- fix stop service return 143
|
||||
|
||||
* Wed Sep 01 2021 wangkai <wangkai385@huawei.com> - 1.2.4-1
|
||||
- Upgrade to 1.2.4 for fix CVE-2021-38294-CVE-2021-40865
|
||||
|
||||
* Fri Aug 20 2021 sunguoshuai <sunguoshuai@huawei.com> - 1.2.3-4
|
||||
- Fix runtime error and uninstall warning
|
||||
|
||||
* Mon Jun 14 2021 lingsheng <lingsheng@huawei.com> - 1.2.3-3
|
||||
- Fix status failure after stopping service
|
||||
|
||||
* Wed May 26 2021 wangchong <wangchong56@huawei.com> - 1.2.3-2
|
||||
- add requires
|
||||
|
||||
* Thu Mar 18 2021 huanghaitao <huanghaitao8@huawei.com> 1.2.3 - 1
|
||||
* Thu Mar 18 2021 huanghaitao <huanghaitao8@huawei.com> - 1.2.3-1
|
||||
- Package init
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user