Linux迷+Python粉 - 技术//blog.pythonwood.com/2019-08-22T14:00:00+08:00面试高频题-LRU缓存的python实现2019-08-22T14:00:00+08:002019-08-22T14:00:00+08:00pythonwoodtag:blog.pythonwood.com,2019-08-22:/2019/08/面试高频题-LRU缓存的python实现/<p>过程简略:&nbsp;url去重的方法,数据库四种隔离级别,乐观锁悲观锁,算法题研讨。算法讨论占了很长时间,以下是把这个过程沉淀后的一遍随笔。</p> <h3 id="leetcode146-lru">leetcode算法题:146. <span class="caps">LRU</span>缓存机制<a class="headerlink" href="#leetcode146-lru" title="Permanent link">&para;</a></h3> <p><a href="https://leetcode-cn.com/problems/lru-cache/">https://leetcode-cn.com/problems/lru-cache/</a></p> <blockquote> <p>运用你所掌握的数据结构,设计和实现一个  <span class="caps">LRU</span> (最近最少使用 …</p></blockquote><p>过程简略:&nbsp;url去重的方法,数据库四种隔离级别,乐观锁悲观锁,算法题研讨。算法讨论占了很长时间,以下是把这个过程沉淀后的一遍随笔。</p> <h3 id="leetcode146-lru">leetcode算法题:146. <span class="caps">LRU</span>缓存机制<a class="headerlink" href="#leetcode146-lru" title="Permanent link">&para;</a></h3> <p><a href="https://leetcode-cn.com/problems/lru-cache/">https://leetcode-cn.com/problems/lru-cache/</a></p> <blockquote> <p>运用你所掌握的数据结构,设计和实现一个  <span class="caps">LRU</span> (最近最少使用) 缓存机制。它应该支持以下操作: 获取数据 get 和 写入数据 put&nbsp;。</p> <p>获取数据 get(key) - 如果密钥 (key) 存在于缓存中,则获取密钥的值(总是正数),否则返回&nbsp;-1。</p> <p>写入数据 put(key, value) -&nbsp;如果密钥不存在,则写入其数据值。当缓存容量达到上限时,它应该在写入新数据之前删除最近最少使用的数据值,从而为新的数据值留出空间。</p> <p>进阶:</p> <p>你是否可以在 O(1)&nbsp;时间复杂度内完成这两种操作?</p> <p>示例:</p> </blockquote> <div class="highlight"><pre><span></span>LRUCache cache = new LRUCache( 2 /* 缓存容量 */ ); cache.put(1, 1); cache.put(2, 2); cache.get(1); // 返回 1 cache.put(3, 3); // 该操作会使得密钥 2 作废 cache.get(2); // 返回 -1 (未找到) cache.put(4, 4); // 该操作会使得密钥 1 作废 cache.get(1); // 返回 -1 (未找到) cache.get(3); // 返回 3 cache.get(4); // 返回 4 </pre></div> <h2 id="_1">思考过程简要<a class="headerlink" href="#_1" title="Permanent link">&para;</a></h2> <p>第一次遇到,没有做好题。之后总结思考如下。面试完重新整理好代码,才通过。</p> <p>1、数据结构知识弱,链表随机增删复杂度O(1),&nbsp;数组复杂度O(n)。使用按时间排序的双向链表。头部总是最新访问或插入的,尾巴总是最老的。使用额外maxsize限制大小,nowsize记录目前大小。</p> <p>2、链表元素是key-val的进一步封装的Node类。链表是另一个类,类总head和tail表示头尾两个空的node。一直不删。</p> <p>3、开头想到最小堆,用得不对。二分法等比较查找下限O(lgN),与哈希法的O(1)差距意识不明显。而哈希函数用内置字典即可。</p> <p>4、拆分小函数复用代码。比如get = 出链 + 插入头部, put = 出链 + 插入头部(update) 或者&nbsp;插入头部(非update)</p> <p>5、之后发现python有functools.lru_cache的现成实现,值得学习。</p> <h2 id="leetcode">leetcode通过提交的代码<a class="headerlink" href="#leetcode" title="Permanent link">&para;</a></h2> <div class="highlight"><pre><span></span><span class="kr">class</span> <span class="nx">Node</span>: <span class="kt">def</span> <span class="nx">__init__</span><span class="p">(</span><span class="nx">self</span><span class="p">,</span> <span class="nx">key</span><span class="o">=</span><span class="nx">None</span><span class="p">,</span> <span class="nx">val</span><span class="o">=</span><span class="nx">None</span><span class="p">)</span><span class="o">:</span> <span class="nx">self</span><span class="p">.</span><span class="nx">key</span> <span class="o">=</span> <span class="nx">key</span> <span class="nx">self</span><span class="p">.</span><span class="nx">val</span> <span class="o">=</span> <span class="nx">val</span> <span class="nx">self</span><span class="p">.</span><span class="nx">prev</span> <span class="o">=</span> <span class="nx">None</span> <span class="nx">self</span><span class="p">.</span><span class="nx">next</span> <span class="o">=</span> <span class="nx">None</span> <span class="kr">class</span> <span class="nx">LRUCache</span><span class="o">:</span> <span class="s1">&#39;&#39;&#39;</span> <span class="s1"> 小函数: 删node,入head</span> <span class="s1"> 注意:head和tail是空的node,不能删除</span> <span class="s1"> &#39;&#39;&#39;</span> <span class="nx">def</span> <span class="nx">__init__</span><span class="p">(</span><span class="nx">self</span><span class="p">,</span> <span class="nx">maxsize</span><span class="o">=</span><span class="mi">10</span><span class="p">)</span><span class="o">:</span> <span class="nx">self</span><span class="p">.</span><span class="nx">maxsize</span> <span class="o">=</span> <span class="nx">maxsize</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nowsize</span> <span class="o">=</span> <span class="mi">0</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span> <span class="o">=</span> <span class="nx">Node</span><span class="p">()</span> <span class="nx">self</span><span class="p">.</span><span class="nx">tail</span> <span class="o">=</span> <span class="nx">Node</span><span class="p">()</span> <span class="nx">self</span><span class="p">.</span><span class="nx">tail</span><span class="p">.</span><span class="nx">prev</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span><span class="p">.</span><span class="nx">next</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">tail</span> <span class="err">#</span> <span class="nx">key</span> <span class="o">-&gt;</span> <span class="nx">nodes</span><span class="p">.</span><span class="nx">val</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nodes</span> <span class="o">=</span> <span class="p">{}</span> <span class="nx">def</span> <span class="nx">get</span><span class="p">(</span><span class="nx">self</span><span class="p">,</span> <span class="nx">key</span><span class="p">)</span><span class="o">:</span> <span class="nx">node</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nodes</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="nx">None</span><span class="p">)</span> <span class="k">if</span> <span class="nx">node</span> <span class="o">!=</span> <span class="nx">None</span><span class="o">:</span> <span class="err">#</span> <span class="mi">2019</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">21</span><span class="err">面试时加的要求</span> <span class="nx">self</span><span class="p">.</span><span class="nx">popNode</span><span class="p">(</span><span class="nx">node</span><span class="p">)</span> <span class="nx">self</span><span class="p">.</span><span class="nx">addNode</span><span class="p">(</span><span class="nx">node</span><span class="p">)</span> <span class="k">return</span> <span class="nx">node</span><span class="p">.</span><span class="nx">val</span> <span class="k">else</span><span class="o">:</span> <span class="k">return</span> <span class="o">-</span><span class="mi">1</span> <span class="nx">def</span> <span class="nx">put</span><span class="p">(</span><span class="nx">self</span><span class="p">,</span> <span class="nx">key</span><span class="p">,</span> <span class="nx">val</span><span class="p">)</span><span class="o">:</span> <span class="nx">self</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="nx">val</span><span class="p">)</span> <span class="nx">def</span> <span class="nx">set</span><span class="p">(</span><span class="nx">self</span><span class="p">,</span> <span class="nx">key</span><span class="p">,</span> <span class="nx">val</span><span class="p">)</span><span class="o">:</span> <span class="nx">node</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nodes</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="nx">None</span><span class="p">)</span> <span class="k">if</span> <span class="nx">node</span> <span class="o">!=</span> <span class="nx">None</span><span class="o">:</span> <span class="err">#</span> <span class="err">在链表中,</span><span class="nx">update</span><span class="err">操作</span> <span class="nx">self</span><span class="p">.</span><span class="nx">popNode</span><span class="p">(</span><span class="nx">node</span><span class="p">)</span> <span class="err">#</span> <span class="nx">self</span><span class="p">.</span><span class="nx">addNode</span><span class="p">(</span><span class="nx">Node</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="nx">val</span><span class="p">))</span> <span class="nx">def</span> <span class="k">delete</span><span class="p">(</span><span class="nx">self</span><span class="p">,</span> <span class="nx">key</span><span class="p">)</span><span class="o">:</span> <span class="nx">node</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nodes</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="nx">key</span><span class="p">,</span> <span class="nx">None</span><span class="p">)</span> <span class="k">if</span> <span class="nx">node</span> <span class="o">!=</span> <span class="nx">None</span>: <span class="kt">self.popNode</span><span class="p">(</span><span class="nx">node</span><span class="p">)</span> <span class="nx">def</span> <span class="nx">popNode</span><span class="p">(</span><span class="nx">self</span><span class="p">,</span> <span class="nx">node</span><span class="p">)</span><span class="o">:</span> <span class="k">if</span> <span class="nx">node</span><span class="p">.</span><span class="nx">val</span> <span class="o">==</span> <span class="nx">None</span>: <span class="kt">return</span> <span class="err">#</span> <span class="err">使用</span><span class="nx">head</span><span class="err">和</span><span class="nx">tail</span><span class="err">为空</span><span class="nx">node</span><span class="err">时无需检查</span> <span class="err">#</span> <span class="k">if</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span><span class="o">:</span> <span class="err">#</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span><span class="p">.</span><span class="nx">prev</span> <span class="o">=</span> <span class="nx">node</span><span class="p">.</span><span class="nx">prev</span> <span class="err">#</span> <span class="k">else</span><span class="o">:</span> <span class="err">#</span> <span class="nx">self</span><span class="p">.</span><span class="nx">tail</span> <span class="o">=</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span> <span class="err">#</span> <span class="k">if</span> <span class="nx">node</span><span class="p">.</span><span class="nx">prev</span><span class="o">:</span> <span class="err">#</span> <span class="nx">node</span><span class="p">.</span><span class="nx">prev</span><span class="p">.</span><span class="nx">next</span> <span class="o">=</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span> <span class="err">#</span> <span class="k">else</span><span class="o">:</span> <span class="err">#</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span> <span class="o">=</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span><span class="p">.</span><span class="nx">prev</span> <span class="o">=</span> <span class="nx">node</span><span class="p">.</span><span class="nx">prev</span> <span class="nx">node</span><span class="p">.</span><span class="nx">prev</span><span class="p">.</span><span class="nx">next</span> <span class="o">=</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nowsize</span> <span class="o">-=</span> <span class="mi">1</span> <span class="err">#</span> <span class="err">字典更新</span> <span class="nx">del</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nodes</span><span class="p">[</span><span class="nx">node</span><span class="p">.</span><span class="nx">key</span><span class="p">]</span> <span class="k">return</span> <span class="nx">node</span> <span class="nx">def</span> <span class="nx">addNode</span><span class="p">(</span><span class="nx">self</span><span class="p">,</span> <span class="nx">node</span><span class="p">)</span><span class="o">:</span> <span class="err">#</span> <span class="err">总在链表头加入</span> <span class="k">if</span> <span class="nx">node</span><span class="p">.</span><span class="nx">val</span> <span class="o">==</span> <span class="nx">None</span>: <span class="kt">return</span> <span class="k">if</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nowsize</span> <span class="o">&gt;=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">maxsize</span><span class="o">:</span> <span class="err">#</span> <span class="err">满了,就去掉尾巴再插入</span> <span class="nx">self</span><span class="p">.</span><span class="nx">popNode</span><span class="p">(</span><span class="nx">self</span><span class="p">.</span><span class="nx">tail</span><span class="p">.</span><span class="nx">prev</span><span class="p">)</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span><span class="p">.</span><span class="nx">next</span> <span class="nx">node</span><span class="p">.</span><span class="nx">prev</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span><span class="p">.</span><span class="nx">next</span><span class="p">.</span><span class="nx">prev</span> <span class="o">=</span> <span class="nx">node</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span><span class="p">.</span><span class="nx">next</span> <span class="o">=</span> <span class="nx">node</span> <span class="err">#</span> <span class="err">费时的</span><span class="nx">debug</span><span class="err">错误:这句之后</span><span class="p">[[</span><span class="s1">&#39;c&#39;</span><span class="p">,</span> <span class="mi">12</span><span class="p">]]</span> <span class="o">-&gt;</span> <span class="p">[]</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nowsize</span> <span class="o">+=</span> <span class="mi">1</span> <span class="err">#</span> <span class="err">字典增加</span> <span class="nx">self</span><span class="p">.</span><span class="nx">nodes</span><span class="p">[</span><span class="nx">node</span><span class="p">.</span><span class="nx">key</span><span class="p">]</span> <span class="o">=</span> <span class="nx">node</span> <span class="kd">@property</span> <span class="nx">def</span> <span class="nx">vals</span><span class="p">(</span><span class="nx">self</span><span class="p">)</span><span class="o">:</span> <span class="err">#</span> <span class="nx">debug</span> <span class="err">#</span> <span class="nx">node</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span> <span class="err">#</span> <span class="err">错误:发现用时</span><span class="mi">15</span><span class="err">分钟</span> <span class="nx">node</span> <span class="o">=</span> <span class="nx">self</span><span class="p">.</span><span class="nx">head</span><span class="p">.</span><span class="nx">next</span> <span class="nx">_vals</span> <span class="o">=</span> <span class="p">[]</span> <span class="err">#</span> <span class="k">while</span> <span class="nx">node</span><span class="p">.</span><span class="nx">val</span><span class="o">:</span> <span class="err">#</span> <span class="mi">2019</span><span class="o">-</span><span class="mi">08</span><span class="o">-</span><span class="mi">21</span><span class="err">大坑,</span><span class="nx">node</span><span class="p">.</span><span class="nx">val</span> <span class="o">=</span> <span class="mi">0</span><span class="err">时会失败!费了</span><span class="mi">1</span><span class="err">小时!!!</span> <span class="k">while</span> <span class="nx">node</span> <span class="o">!=</span> <span class="nx">None</span> <span class="nx">and</span> <span class="nx">node</span><span class="p">.</span><span class="nx">val</span> <span class="o">!=</span> <span class="nx">None</span>: <span class="kt">_vals.append</span><span class="p">([</span><span class="nx">node</span><span class="p">.</span><span class="nx">key</span><span class="p">,</span><span class="nx">node</span><span class="p">.</span><span class="nx">val</span><span class="p">])</span> <span class="nx">node</span> <span class="o">=</span> <span class="nx">node</span><span class="p">.</span><span class="nx">next</span> <span class="k">return</span> <span class="nx">_vals</span> <span class="nx">def</span> <span class="nx">pprint</span><span class="p">(</span><span class="nx">self</span><span class="p">)</span><span class="o">:</span> <span class="err">#</span> <span class="nx">debug</span> <span class="nx">print</span><span class="p">(</span><span class="nx">self</span><span class="p">.</span><span class="nx">vals</span><span class="p">,</span> <span class="nx">flush</span><span class="o">=</span><span class="nx">True</span><span class="p">)</span> <span class="k">if</span> <span class="nx">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="o">:</span> <span class="nx">lru</span> <span class="o">=</span> <span class="nx">LRUCache</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">&#39;b&#39;</span><span class="p">,</span> <span class="mi">12</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">pprint</span><span class="p">()</span> <span class="nx">print</span><span class="p">(</span><span class="s1">&#39;stage end: -----------------&#39;</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">&#39;a&#39;</span><span class="p">,</span> <span class="mi">12</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">&#39;d&#39;</span><span class="p">,</span> <span class="mi">12</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">&#39;c&#39;</span><span class="p">,</span> <span class="mi">12</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">pprint</span><span class="p">()</span> <span class="nx">print</span><span class="p">(</span><span class="s1">&#39;stage end: -----------------&#39;</span><span class="p">)</span> <span class="nx">print</span><span class="p">(</span><span class="nx">lru</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s1">&#39;a&#39;</span><span class="p">))</span> <span class="nx">print</span><span class="p">(</span><span class="s1">&#39;stage end: -----------------&#39;</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="k">delete</span><span class="p">(</span><span class="s1">&#39;a&#39;</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">pprint</span><span class="p">()</span> <span class="nx">print</span><span class="p">(</span><span class="s1">&#39;stage end: -----------------&#39;</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s1">&#39;d&#39;</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="nx">lru</span><span class="p">.</span><span class="nx">pprint</span><span class="p">()</span> <span class="nx">print</span><span class="p">(</span><span class="s1">&#39;stage end: -----------------&#39;</span><span class="p">)</span> <span class="err">#</span> <span class="nx">leetcode</span><span class="err">的测试用例</span> <span class="nx">cache</span> <span class="o">=</span> <span class="nx">LRUCache</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">put</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">put</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">);</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span> <span class="err">#</span> <span class="err">返回</span> <span class="mi">1</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">pprint</span><span class="p">()</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">put</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">);</span> <span class="err">#</span> <span class="err">该操作会使得密钥</span> <span class="mi">2</span> <span class="err">作废</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">pprint</span><span class="p">()</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="mi">2</span><span class="p">);</span> <span class="err">#</span> <span class="err">返回</span> <span class="o">-</span><span class="mi">1</span> <span class="p">(</span><span class="err">未找到</span><span class="p">)</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">pprint</span><span class="p">()</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">put</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">);</span> <span class="err">#</span> <span class="err">该操作会使得密钥</span> <span class="mi">1</span> <span class="err">作废</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span> <span class="err">#</span> <span class="err">返回</span> <span class="o">-</span><span class="mi">1</span> <span class="p">(</span><span class="err">未找到</span><span class="p">)</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">pprint</span><span class="p">()</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="mi">3</span><span class="p">);</span> <span class="err">#</span> <span class="err">返回</span> <span class="mi">3</span> <span class="nx">cache</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="mi">4</span><span class="p">);</span> <span class="err">#</span> <span class="err">返回</span> <span class="mi">4</span> </pre></div> <h2 id="_2">参考<a class="headerlink" href="#_2" title="Permanent link">&para;</a></h2> <ol> <li> <p>leetcode算法题:146. <span class="caps">LRU</span>缓存机制 <a href="https://leetcode-cn.com/problems/lru-cache/">https://leetcode-cn.com/problems/lru-cache/</a></p> </li> <li> <p>Python 缓存机制与 functools.lru_cache <a href="http://blog.konghy.cn/2016/04/20/python-cache/">http://blog.konghy.cn/2016/04/20/python-cache/</a></p> </li> </ol>Linux服务器安装轻量X环境xfce桌面和VNC服务2019-04-27T21:00:00+08:002019-04-27T21:00:00+08:00pythonwoodtag:blog.pythonwood.com,2019-04-27:/2019/04/Linux服务器安装轻量X环境xfce桌面和VNC服务/<p>有些任务需要Linux桌面环境,例如使用chrome/firefox浏览器进行测试或抓取数据。简要记录安装过程备需。任何时候,一个免key的<span class="caps">SSH</span>登录环境都能带来方便。</p> <p>以下环境是Ubuntu18.04,&nbsp;Ubuntu其他版本大同小异。其他linux发行版需修改。</p> <h3 id="xfce">安装必须的xfce桌面基础包。还有语言支持包<a class="headerlink" href="#xfce" title="Permanent link">&para;</a></h3> <p>桌面环境我选xfce。足够轻量体验也很好。vnc服务器我选tightvncserver,简单高效。</p> <div class="highlight"><pre><span></span># X环境,设置中文环境 …</pre></div><p>有些任务需要Linux桌面环境,例如使用chrome/firefox浏览器进行测试或抓取数据。简要记录安装过程备需。任何时候,一个免key的<span class="caps">SSH</span>登录环境都能带来方便。</p> <p>以下环境是Ubuntu18.04,&nbsp;Ubuntu其他版本大同小异。其他linux发行版需修改。</p> <h3 id="xfce">安装必须的xfce桌面基础包。还有语言支持包<a class="headerlink" href="#xfce" title="Permanent link">&para;</a></h3> <p>桌面环境我选xfce。足够轻量体验也很好。vnc服务器我选tightvncserver,简单高效。</p> <div class="highlight"><pre><span></span># X环境,设置中文环境 sudo apt install xfdesktop4 tightvncserver xfce4-terminal xfce4-panel ttf-wqy-zenhei ttf-wqy-microhei language-pack-zh-hans-base language-pack-zh-hans </pre></div> <h3 id="_1">设置桌面显示中文<a class="headerlink" href="#_1" title="Permanent link">&para;</a></h3> <p>英语普通4级的我,桌面环境还是用母语熟悉。</p> <div class="highlight"><pre><span></span># 设置中文环境 sudo dpkg-reconfigure locales # 时区 sudo dpkg-reconfigure tzdata </pre></div> <h3 id="vncxstartup">初始化vnc,设置密码和xstartup环境<a class="headerlink" href="#vncxstartup" title="Permanent link">&para;</a></h3> <p>完成后相关配置文件都在$<span class="caps">HOME</span>/.vnc文件夹内,复制文件夹设置权限归属即可,不须重新配置。</p> <h4 id="vnc">初始一个默认.vnc文件夹<a class="headerlink" href="#vnc" title="Permanent link">&para;</a></h4> <div class="highlight"><pre><span></span>vncserver -localhost :1 # 开启1号,只监听在127.0.0.1本地lo网卡 # 监听0.0.0.0可用 vncserver :1 vncserver -kill :1 # 关闭1号 </pre></div> <h4 id="vncxstartupvnc">修改~/.vnc/xstartup,登录vnc后的启动桌面命令<a class="headerlink" href="#vncxstartupvnc" title="Permanent link">&para;</a></h4> <p>文件最后一句内容</p> <div class="highlight"><pre><span></span>startxfce4 &amp; </pre></div> <h4 id="vncpasswdvnc">使用vncpasswd命令设置vnc密码<a class="headerlink" href="#vncpasswdvnc" title="Permanent link">&para;</a></h4> <div class="highlight"><pre><span></span>vncpasswd </pre></div> <h3 id="vnc_1">vnc启动与安全<a class="headerlink" href="#vnc_1" title="Permanent link">&para;</a></h3> <p>参考上面即可。如监听localhost地址,可用ssh本地转发功能建立ssh隧道再进行连接。</p> <div class="highlight"><pre><span></span>ssh -gfTNL 5901:localhost:5901 host # vnc的1号端口对应5901, 2、3等端口对应递增。 </pre></div> <h3 id="vnc_2">vnc服务要开机启动<a class="headerlink" href="#vnc_2" title="Permanent link">&para;</a></h3> <p>因版本不同差别较大,需要配置较多,vnc服务不是常用,不需开机自启。</p> <h2 id="_2">参考<a class="headerlink" href="#_2" title="Permanent link">&para;</a></h2> <p>How to Install and Configure <span class="caps">VNC</span> on Ubuntu <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-16-04">https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-vnc-on-ubuntu-16-04</a></p> <p>How to install <span class="caps">VNC</span> on Linux ( <span class="caps">GUI</span> for your Linux <span class="caps">VPS</span> ) <a href="https://www.interserver.net/tips/kb/install-vnc-linux-gui-linux-vps/">https://www.interserver.net/tips/kb/install-vnc-linux-gui-linux-vps/</a></p>Linux释放磁盘空间——系统日志systemd-journal清理2018-12-05T15:00:00+08:002018-12-05T15:00:00+08:00pythonwoodtag:blog.pythonwood.com,2018-12-05:/2018/12/Linux释放磁盘空间——系统日志systemd-journal清理/<h2 id="varlogjournal">/var/log/journal 目录占用空间很大<a class="headerlink" href="#varlogjournal" title="Permanent link">&para;</a></h2> <p>原因systemd系统通过systemd-journald.service记录日志.&nbsp;默认以二进制写入/var/log/journal/目录中的日志文件,系统安装久了发现磁盘空间逐渐变小。</p> <p>ubuntu18.04,&nbsp;centos7等新系统都使用新型系统systemd,就可能需要清理。</p> <div class="highlight"><pre><span></span>$ du -sh /var/log …</pre></div><h2 id="varlogjournal">/var/log/journal 目录占用空间很大<a class="headerlink" href="#varlogjournal" title="Permanent link">&para;</a></h2> <p>原因systemd系统通过systemd-journald.service记录日志.&nbsp;默认以二进制写入/var/log/journal/目录中的日志文件,系统安装久了发现磁盘空间逐渐变小。</p> <p>ubuntu18.04,&nbsp;centos7等新系统都使用新型系统systemd,就可能需要清理。</p> <div class="highlight"><pre><span></span>$ du -sh /var/log/journal/ <span class="m">2</span>.2G /var/log/journal/ </pre></div> <h2 id="_1">手动命令行清理 单次生效 可临时救急<a class="headerlink" href="#_1" title="Permanent link">&para;</a></h2> <p>删除数天以前旧日志</p> <div class="highlight"><pre><span></span># journalctl --vacuum-time=7d Vacuuming done, freed 2.1G of archived journals from /var/log/journal/1095e22a7289463f9f4fdd6d10e3da34. </pre></div> <p>删除到只保留100M日志量的状态</p> <div class="highlight"><pre><span></span># journalctl --vacuum-size=100M Vacuuming done, freed 2.0G of archived journals from /var/log/journal/1095e22a7289463f9f4fdd6d10e3da34. </pre></div> <h2 id="systemd-journaldservice">配置systemd-journald.service 永久生效<a class="headerlink" href="#systemd-journaldservice" title="Permanent link">&para;</a></h2> <p>systemd-journald 的配置文件为 /etc/systemd/journald.conf&nbsp;中,将SystemMaxUse=这行去掉注释,修改为SystemMaxUse=1G,将日志总量限制在1G内。</p> <p>预计下次启动生效,但我测试暂没有效果,不知原因。</p> <h2 id="_2">参考<a class="headerlink" href="#_2" title="Permanent link">&para;</a></h2> <p>clear up systemd-journal <a href="https://ma.ttias.be/clear-systemd-journal/">https://ma.ttias.be/clear-systemd-journal/</a></p> <p>使用journalctl查看systemd日志 <a href="https://lujun9972.github.io/blog/2018/08/08/使用journalctl查看systemd日志/">https://lujun9972.github.io/blog/2018/08/08/使用journalctl查看systemd日志/</a></p>网络检测ping脚本——搬瓦工CN2机房PING测试2018-11-25T20:00:00+08:002018-11-27T16:00:00+08:00pythonwoodtag:blog.pythonwood.com,2018-11-25:/2018/11/网络检测ping脚本——搬瓦工CN2机房PING测试/<h2 id="vps">项目起源:搬瓦工vps快过期<a class="headerlink" href="#vps" title="Permanent link">&para;</a></h2> <p>为决定是否upgrade到cn2机房,需要先验证<span class="caps">CN2</span>机房网络是不是更好。一直没有一个适合自己的ping工具,自己写一个。</p> <h3 id="openwrt5ping">部署在openwrt路由,定时任务每5分钟启动ping一次<a class="headerlink" href="#openwrt5ping" title="Permanent link">&para;</a></h3> <p>脚本兼容新旧版本ping(输出内容不同),使用sh标准达到最大linux兼容。已兼容openwrt。</p> <div class="highlight"><pre><span></span># ping 50次,不超过90s,丢包率每字符表示2%,延迟每字符表示10ms */5 …</pre></div><h2 id="vps">项目起源:搬瓦工vps快过期<a class="headerlink" href="#vps" title="Permanent link">&para;</a></h2> <p>为决定是否upgrade到cn2机房,需要先验证<span class="caps">CN2</span>机房网络是不是更好。一直没有一个适合自己的ping工具,自己写一个。</p> <h3 id="openwrt5ping">部署在openwrt路由,定时任务每5分钟启动ping一次<a class="headerlink" href="#openwrt5ping" title="Permanent link">&para;</a></h3> <p>脚本兼容新旧版本ping(输出内容不同),使用sh标准达到最大linux兼容。已兼容openwrt。</p> <div class="highlight"><pre><span></span># ping 50次,不超过90s,丢包率每字符表示2%,延迟每字符表示10ms */5 * * * * /path/to/20181119_host_ping.sh 23.252.103.101 50 90 2 10 &gt;&gt;/tmp/ping.DC3.txt 2&gt;/dev/null */5 * * * * /path/to/20181119_host_ping.sh 65.49.131.102 50 90 2 10 &gt;&gt;/tmp/ping.DC9.txt 2&gt;/dev/null */5 * * * * /path/to/20181119_host_ping.sh vps.old 50 90 2 10 &gt;&gt;/tmp/ping.vps.old.txt 2&gt;/dev/null */5 * * * * /path/to/20181119_host_ping.sh vps.new 50 90 2 10 &gt;&gt;/tmp/ping.vps.new.txt 2&gt;/dev/null </pre></div> <h2 id="host">最终版本的脚本,单host版<a class="headerlink" href="#host" title="Permanent link">&para;</a></h2> <h3 id="_1">脚本使用与输出字段意义<a class="headerlink" href="#_1" title="Permanent link">&para;</a></h3> <div class="highlight"><pre><span></span>$ ./20181119_host_ping.sh 请参考文档使用: ./20181119_host_ping.sh &lt;host<span class="p">|</span><span class="m">163</span>.com&gt; &lt;pkgcn<span class="p">|</span><span class="m">10</span>&gt; &lt;maxtime<span class="p">|</span><span class="m">30</span>&gt; &lt;lost_step<span class="p">|</span><span class="m">2</span>&gt; &lt;rt_step<span class="p">|</span><span class="m">5</span>&gt; ./20181119_host_ping.sh <span class="m">163</span>.com <span class="m">10</span> <span class="m">30</span> <span class="m">1</span> <span class="m">5</span> <span class="o">(</span>测10次ping,不超过30s,丢包率每字符表示1%,延迟每字符表示5ms<span class="o">)</span> 时间 出包 入包 丢包 min avg max <span class="p">|</span>图形化丢包率,最长25字符 <span class="p">|</span>图形化延迟avg,最多50字符 <span class="m">2018</span>-11-20_10:08:55 <span class="m">3</span> <span class="m">3</span> <span class="m">2</span>% <span class="m">166</span> <span class="m">167</span> <span class="m">169</span> <span class="p">|</span>xx <span class="p">|</span>**************** </pre></div> <h2 id="_2">比较结果<a class="headerlink" href="#_2" title="Permanent link">&para;</a></h2> <p>搬瓦工<span class="caps">CN2</span> <span class="caps">GIA</span>质量非常稳定,土豪随意;对于移动用户访问,<span class="caps">CN2</span>比一般机房提升不大。</p> <p>一天之中2-10点网络最好,晚上6点至凌晨1点最差</p> <div class="highlight"><pre><span></span>root@Gargoyle:/tmp# tail ping.DC9.txt ping.DC3.txt ping.vps.old.txt -n 50 ==&gt; ping.DC9.txt &lt;== 16:00:01 50 50 0% 163 166 170 | |**************** 16:05:01 51 50 1% 163 177 301 | |***************** 16:10:01 50 50 0% 162 166 170 | |**************** 16:15:01 50 50 0% 164 166 171 | |**************** 16:20:01 50 50 0% 164 166 169 | |**************** 16:25:01 50 50 0% 162 166 171 | |**************** 16:30:02 50 50 0% 164 166 171 | |**************** 16:35:01 50 50 0% 163 167 174 | |**************** 16:40:02 50 50 0% 164 167 172 | |**************** 16:45:01 50 50 0% 164 168 173 | |**************** 16:50:01 50 50 0% 163 166 171 | |**************** 16:55:01 50 50 0% 163 166 170 | |**************** 17:00:01 51 50 1% 163 166 174 | |**************** 17:05:01 50 50 0% 164 167 178 | |**************** 17:10:01 50 50 0% 163 166 171 | |**************** 17:15:01 50 50 0% 163 166 170 | |**************** 17:20:01 50 50 0% 163 166 169 | |**************** 17:25:01 50 50 0% 163 166 169 | |**************** 17:30:01 50 50 0% 163 166 171 | |**************** 17:35:01 50 50 0% 163 166 170 | |**************** 17:40:01 50 50 0% 164 167 172 | |**************** 17:45:01 50 50 0% 164 167 169 | |**************** 17:50:01 50 50 0% 164 166 169 | |**************** 17:55:01 50 50 0% 162 166 171 | |**************** 18:00:01 50 50 0% 164 166 170 | |**************** 18:05:01 50 50 0% 164 167 170 | |**************** 18:10:01 50 50 0% 163 165 168 | |**************** 18:15:01 50 50 0% 164 166 170 | |**************** 18:20:01 50 50 0% 164 166 170 | |**************** 18:25:01 50 50 0% 164 166 169 | |**************** 18:30:01 50 50 0% 164 166 169 | |**************** 18:35:01 50 50 0% 164 166 171 | |**************** 18:40:01 50 50 0% 164 167 171 | |**************** 18:45:01 50 50 0% 163 166 170 | |**************** 18:50:01 50 50 0% 164 166 171 | |**************** 18:55:01 50 50 0% 164 166 171 | |**************** 19:00:01 50 50 0% 164 167 188 | |**************** 19:05:01 50 50 0% 164 168 188 | |**************** 19:10:01 50 50 0% 163 166 171 | |**************** 19:15:01 50 50 0% 164 169 174 | |**************** 19:20:01 50 50 0% 163 168 183 | |**************** 19:25:01 50 50 0% 164 167 180 | |**************** 19:30:01 50 50 0% 164 168 187 | |**************** 19:35:01 50 50 0% 163 169 193 | |**************** 19:40:01 50 50 0% 162 168 196 | |**************** 19:45:01 50 50 0% 163 166 169 | |**************** 19:50:01 50 50 0% 163 167 169 | |**************** 19:55:01 50 50 0% 163 166 170 | |**************** 20:00:01 50 50 0% 163 167 192 | |**************** 20:05:01 50 50 0% 162 166 171 | |**************** ==&gt; ping.DC3.txt &lt;== 16:00:01 59 50 15% 234 283 333 |xxxxxxx |**************************** 16:05:01 56 50 10% 174 241 318 |xxxxx |************************ 16:10:01 52 50 3% 188 246 264 |x |************************ 16:15:01 51 50 1% 179 229 255 | |********************** 16:20:02 50 50 0% 189 245 261 | |************************ 16:25:01 51 50 1% 188 246 267 | |************************ 16:30:02 52 50 3% 187 245 265 |x |************************ 16:35:02 50 50 0% 212 249 262 | |************************ 16:40:02 52 50 3% 181 249 264 |x |************************ 16:45:01 53 50 5% 191 233 261 |xx |*********************** 16:50:01 53 50 5% 188 249 263 |xx |************************ 16:55:01 50 50 0% 174 241 263 | |************************ 17:00:01 50 50 0% 208 248 263 | |************************ 17:05:01 50 50 0% 204 242 262 | |************************ 17:10:01 54 50 7% 188 216 252 |xxx |********************* 17:15:01 51 50 1% 218 252 263 | |************************* 17:20:01 51 50 1% 205 249 265 | |************************ 17:25:01 51 50 1% 196 243 261 | |************************ 17:30:01 52 50 3% 197 249 265 |x |************************ 17:35:01 52 50 3% 224 249 262 |x |************************ 17:40:01 54 50 7% 213 251 266 |xxx |************************* 17:45:01 50 50 0% 221 247 265 | |************************ 17:50:01 52 50 3% 206 246 260 |x |************************ 17:55:01 53 50 5% 180 241 265 |xx |************************ 18:00:01 50 50 0% 207 243 262 | |************************ 18:05:01 52 50 3% 198 243 263 |x |************************ 18:10:01 52 50 3% 197 252 264 |x |************************* 18:15:01 51 50 1% 220 247 263 | |************************ 18:20:01 51 50 1% 205 245 263 | |************************ 18:25:01 50 50 0% 178 221 247 | |********************** 18:30:01 51 50 1% 178 237 262 | |*********************** 18:35:01 51 50 1% 161 187 261 | |****************** 18:40:01 51 50 1% 206 241 261 | |************************ 18:45:01 51 50 1% 165 188 209 | |****************** 18:50:01 50 50 0% 209 247 264 | |************************ 18:55:01 50 50 0% 196 236 261 | |*********************** 19:00:01 52 50 3% 167 200 222 |x |******************** 19:05:01 52 50 3% 170 200 238 |x |******************** 19:10:01 50 50 0% 225 248 263 | |************************ 19:15:01 51 50 1% 204 246 263 | |************************ 19:20:01 50 50 0% 226 248 263 | |************************ 19:25:01 51 50 1% 204 249 264 | |************************ 19:30:01 50 50 0% 212 253 265 | |************************* 19:35:01 52 50 3% 206 245 265 |x |************************ 19:40:01 50 50 0% 206 241 286 | |************************ 19:45:01 50 50 0% 180 236 262 | |*********************** 19:50:01 50 50 0% 197 241 264 | |************************ 19:55:01 51 50 1% 213 255 265 | |************************* 20:00:01 54 50 7% 204 254 264 |xxx |************************* 20:05:01 53 50 5% 216 250 265 |xx |************************* ==&gt; ping.vps.old.txt &lt;== 16:00:01 67 50 25% 240 285 338 |xxxxxxxxxxxx |**************************** 16:05:01 61 50 18% 183 247 325 |xxxxxxxxx |************************ 16:10:01 51 50 1% 219 254 270 | |************************* 16:15:02 55 50 9% 182 236 262 |xxxx |*********************** 16:20:01 51 50 1% 197 258 524 | |************************* 16:25:01 50 50 0% 193 276 1166 | |*************************** 16:30:02 52 50 3% 191 250 269 |x |************************* 16:35:02 51 50 1% 231 256 270 | |************************* 16:40:02 50 50 0% 190 254 269 | |************************* 16:45:01 51 50 1% 194 239 266 | |*********************** 16:50:01 50 50 0% 191 256 270 | |************************* 16:55:01 50 50 0% 179 247 266 | |************************ 17:00:01 51 50 1% 220 254 269 | |************************* 17:05:01 50 50 0% 211 248 267 | |************************ 17:10:01 51 50 1% 194 222 256 | |********************** 17:15:01 51 50 1% 213 258 271 | |************************* 17:20:01 52 50 3% 209 255 268 |x |************************* 17:25:01 50 50 0% 199 248 268 | |************************ 17:30:01 50 50 0% 198 255 270 | |************************* 17:35:01 51 50 1% 231 255 270 | |************************* 17:40:01 52 50 3% 220 257 270 |x |************************* 17:45:01 51 50 1% 225 252 267 | |************************* 17:50:01 50 50 0% 208 251 268 | |************************* 17:55:01 51 50 1% 186 248 269 | |************************ 18:00:01 51 50 1% 211 248 272 | |************************ 18:05:01 52 50 3% 207 249 267 |x |************************ 18:10:01 52 50 3% 199 258 269 |x |************************* 18:15:01 50 50 0% 223 252 269 | |************************* 18:20:01 50 50 0% 210 250 267 | |************************* 18:25:01 51 50 1% 184 227 254 | |********************** 18:30:01 50 50 0% 187 244 267 | |************************ 18:35:01 51 50 1% 169 194 268 | |******************* 18:40:01 51 50 1% 212 247 266 | |************************ 18:45:01 50 50 0% 170 193 213 | |******************* 18:50:01 51 50 1% 216 252 269 | |************************* 18:55:01 50 50 0% 201 242 267 | |************************ 19:00:01 50 50 0% 170 206 227 | |******************** 19:05:01 51 50 1% 175 208 261 | |******************** 19:10:01 50 50 0% 230 253 267 | |************************* 19:15:01 53 50 5% 210 253 268 |xx |************************* 19:20:01 50 50 0% 230 254 267 | |************************* 19:25:01 50 50 0% 211 254 268 | |************************* 19:30:01 50 50 0% 221 259 269 | |************************* 19:35:01 51 50 1% 213 252 269 | |************************* 19:40:01 50 50 0% 220 248 268 | |************************ 19:45:01 50 50 0% 187 241 267 | |************************ 19:50:01 50 50 0% 202 247 270 | |************************ 19:55:01 52 50 3% 216 261 268 |x |************************** 20:00:01 51 50 1% 212 258 269 | |************************* 20:05:01 52 50 3% 228 257 270 |x |************************* </pre></div> <h3 id="2018-11-27">2018-11-27后注:<a class="headerlink" href="#2018-11-27" title="Permanent link">&para;</a></h3> <p>作为搬瓦工多年老用户,随着去年以来ssh操作vps卡顿日渐严重,差点就因此粉转黑。&nbsp;不过对我来说,还是搬瓦工性价比高,操作熟悉。</p> <p>非常遗憾,错过史上最大促销(<span class="caps">CPU</span>:2 核 内存:2048 <span class="caps">MB</span> 硬盘:40 <span class="caps">GB</span> <span class="caps">SSD</span> 流量:1 <span class="caps">TB</span> <span class="caps">CN2</span>网络)相当于打3折。&nbsp;25号因为黑五优惠,买了普通cn2机房vps。</p> <p>让人满意的是,ping结果事实证明,新vps比老vps的网络环境好了小一倍。 老旧vps都安装了nginx,使用http播放mp4视频,新vps体验完胜老vps。 就算去掉部分优化的影响:新vps系统是ubuntu18.04并开启了<span class="caps">BBR</span>拥塞算法。</p> <p><strong>最终结果:cn2机房vps完胜老vps</strong></p> <div class="highlight"><pre><span></span>$ tail /tmp/ping.vps* -n <span class="m">40</span> <span class="p">|</span> sed <span class="s1">&#39;s/2018-11-27_//&#39;</span> <span class="p">|</span> sed <span class="s1">&#39;/^16:/d&#39;</span> <span class="o">==</span>&gt; /tmp/ping.vpsold.txt &lt;<span class="o">==</span> <span class="m">13</span>:30:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">180</span> <span class="m">221</span> <span class="m">267</span> <span class="p">|</span> <span class="p">|</span>********************** <span class="m">13</span>:35:01 <span class="m">51</span> <span class="m">50</span> <span class="m">1</span>% <span class="m">216</span> <span class="m">271</span> <span class="m">518</span> <span class="p">|</span> <span class="p">|</span>*************************** <span class="m">13</span>:40:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">222</span> <span class="m">233</span> <span class="m">247</span> <span class="p">|</span> <span class="p">|</span>*********************** <span class="m">13</span>:45:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">195</span> <span class="m">236</span> <span class="m">269</span> <span class="p">|</span> <span class="p">|</span>*********************** <span class="m">13</span>:50:01 <span class="m">55</span> <span class="m">50</span> <span class="m">9</span>% <span class="m">235</span> <span class="m">246</span> <span class="m">265</span> <span class="p">|</span>xxxx <span class="p">|</span>************************ <span class="m">13</span>:55:01 <span class="m">56</span> <span class="m">50</span> <span class="m">10</span>% <span class="m">261</span> <span class="m">281</span> <span class="m">311</span> <span class="p">|</span>xxxxx <span class="p">|</span>**************************** <span class="m">14</span>:00:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">166</span> <span class="m">225</span> <span class="m">248</span> <span class="p">|</span> <span class="p">|</span>********************** <span class="m">14</span>:05:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">168</span> <span class="m">197</span> <span class="m">230</span> <span class="p">|</span> <span class="p">|</span>******************* <span class="m">14</span>:10:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">162</span> <span class="m">181</span> <span class="m">217</span> <span class="p">|</span> <span class="p">|</span>****************** <span class="m">14</span>:15:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">196</span> <span class="m">231</span> <span class="m">280</span> <span class="p">|</span> <span class="p">|</span>*********************** <span class="m">14</span>:20:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">212</span> <span class="m">239</span> <span class="m">265</span> <span class="p">|</span> <span class="p">|</span>*********************** <span class="m">14</span>:25:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">195</span> <span class="m">246</span> <span class="m">288</span> <span class="p">|</span> <span class="p">|</span>************************ <span class="m">14</span>:30:02 <span class="m">51</span> <span class="m">50</span> <span class="m">1</span>% <span class="m">192</span> <span class="m">233</span> <span class="m">263</span> <span class="p">|</span> <span class="p">|</span>*********************** <span class="m">14</span>:35:02 <span class="m">52</span> <span class="m">50</span> <span class="m">3</span>% <span class="m">220</span> <span class="m">264</span> <span class="m">292</span> <span class="p">|</span>x <span class="p">|</span>************************** <span class="m">14</span>:40:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">178</span> <span class="m">211</span> <span class="m">259</span> <span class="p">|</span> <span class="p">|</span>********************* <span class="m">14</span>:45:02 <span class="m">51</span> <span class="m">50</span> <span class="m">1</span>% <span class="m">172</span> <span class="m">204</span> <span class="m">234</span> <span class="p">|</span> <span class="p">|</span>******************** <span class="m">14</span>:50:01 <span class="m">51</span> <span class="m">50</span> <span class="m">1</span>% <span class="m">245</span> <span class="m">281</span> <span class="m">321</span> <span class="p">|</span> <span class="p">|</span>**************************** <span class="m">14</span>:55:01 <span class="m">53</span> <span class="m">50</span> <span class="m">5</span>% <span class="m">231</span> <span class="m">296</span> <span class="m">337</span> <span class="p">|</span>xx <span class="p">|</span>***************************** <span class="m">15</span>:00:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">231</span> <span class="m">263</span> <span class="m">290</span> <span class="p">|</span> <span class="p">|</span>************************** <span class="m">15</span>:05:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">229</span> <span class="m">258</span> <span class="m">282</span> <span class="p">|</span> <span class="p">|</span>************************* <span class="m">15</span>:10:01 <span class="m">51</span> <span class="m">50</span> <span class="m">1</span>% <span class="m">195</span> <span class="m">241</span> <span class="m">277</span> <span class="p">|</span> <span class="p">|</span>************************ <span class="m">15</span>:15:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">213</span> <span class="m">263</span> <span class="m">320</span> <span class="p">|</span> <span class="p">|</span>************************** <span class="m">15</span>:20:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">182</span> <span class="m">208</span> <span class="m">225</span> <span class="p">|</span> <span class="p">|</span>******************** <span class="m">15</span>:25:01 <span class="m">51</span> <span class="m">50</span> <span class="m">1</span>% <span class="m">227</span> <span class="m">268</span> <span class="m">313</span> <span class="p">|</span> <span class="p">|</span>************************** <span class="m">15</span>:30:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">218</span> <span class="m">288</span> <span class="m">340</span> <span class="p">|</span> <span class="p">|</span>**************************** <span class="m">15</span>:35:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">187</span> <span class="m">225</span> <span class="m">339</span> <span class="p">|</span> <span class="p">|</span>********************** <span class="m">15</span>:40:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">224</span> <span class="m">263</span> <span class="m">297</span> <span class="p">|</span> <span class="p">|</span>************************** <span class="m">15</span>:45:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">235</span> <span class="m">259</span> <span class="m">288</span> <span class="p">|</span> <span class="p">|</span>************************* <span class="m">15</span>:50:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">195</span> <span class="m">249</span> <span class="m">274</span> <span class="p">|</span> <span class="p">|</span>************************ <span class="m">15</span>:55:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">220</span> <span class="m">264</span> <span class="m">302</span> <span class="p">|</span> <span class="p">|</span>************************** <span class="o">==</span>&gt; /tmp/ping.vpsold.txt &lt;<span class="o">==</span> <span class="m">13</span>:30:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">156</span> <span class="m">159</span> <span class="m">164</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">13</span>:35:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">13</span>:40:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">156</span> <span class="m">160</span> <span class="m">164</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">13</span>:45:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">13</span>:50:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">159</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">13</span>:55:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">155</span> <span class="m">159</span> <span class="m">163</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">14</span>:00:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">14</span>:05:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">164</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">14</span>:10:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">164</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">14</span>:15:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">159</span> <span class="m">163</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">14</span>:20:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">163</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">14</span>:25:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">163</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">14</span>:30:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">156</span> <span class="m">159</span> <span class="m">163</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">14</span>:35:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">14</span>:40:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">159</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">14</span>:45:02 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">159</span> <span class="m">163</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">14</span>:50:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">163</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">14</span>:55:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">164</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">15</span>:00:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">159</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">15</span>:05:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">15</span>:10:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">159</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">15</span>:15:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">156</span> <span class="m">160</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">15</span>:20:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">156</span> <span class="m">159</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">15</span>:25:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">164</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">15</span>:30:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">166</span> <span class="m">192</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">15</span>:35:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">160</span> <span class="m">190</span> <span class="m">201</span> <span class="p">|</span> <span class="p">|</span>******************* <span class="m">15</span>:40:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">160</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>**************** <span class="m">15</span>:45:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">155</span> <span class="m">159</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">15</span>:50:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">157</span> <span class="m">159</span> <span class="m">162</span> <span class="p">|</span> <span class="p">|</span>*************** <span class="m">15</span>:55:01 <span class="m">50</span> <span class="m">50</span> <span class="m">0</span>% <span class="m">156</span> <span class="m">160</span> <span class="m">176</span> <span class="p">|</span> <span class="p">|</span>**************** </pre></div>神奇的环境bug导致python3中出现udc开头字符串2018-11-07T15:30:00+08:002018-11-07T15:30:00+08:00pythonwoodtag:blog.pythonwood.com,2018-11-07:/2018/11/神奇的环境bug导致python3中出现udc开头字符串/<h2 id="langzh_cnutf-8langen_usutf-8">注意:<span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8与<span class="caps">LANG</span>=en_US.<span class="caps">UTF</span>-8不可混淆!<a class="headerlink" href="#langzh_cnutf-8langen_usutf-8" title="Permanent link">&para;</a></h2> <p><strong><span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8与<span class="caps">LANG</span>=en_US.<span class="caps">UTF</span>-8有区别</strong> ,&nbsp;所以不可混淆!想之前在python2时代吃过坑,没想到到了统一unicode的python3 …</p><h2 id="langzh_cnutf-8langen_usutf-8">注意:<span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8与<span class="caps">LANG</span>=en_US.<span class="caps">UTF</span>-8不可混淆!<a class="headerlink" href="#langzh_cnutf-8langen_usutf-8" title="Permanent link">&para;</a></h2> <p><strong><span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8与<span class="caps">LANG</span>=en_US.<span class="caps">UTF</span>-8有区别</strong> ,&nbsp;所以不可混淆!想之前在python2时代吃过坑,没想到到了统一unicode的python3,因环境不一致也能导致编码问题!</p> <h2 id="_1">当时环境与功能:<a class="headerlink" href="#_1" title="Permanent link">&para;</a></h2> <p>vps系统是ubutnu 14.04, 相关软件python3.4, selenium3+, chrome66, chromedriver。使用crontab启动shell, shell中启动python脚本,&nbsp;脚本中selenium启动chrome,……</p> <h2 id="bug">出bug的运行流程:<a class="headerlink" href="#bug" title="Permanent link">&para;</a></h2> <ol> <li>crontab中的a.sh启动 <strong><span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8 bash&nbsp;a.sh</strong></li> <li>a.sh末尾调用&rdquo;b中文名.py&rdquo;,&nbsp;带中文参数&rdquo;《xxx》&rdquo;</li> <li>b中文.py 中print(参数1) 会异常显示字符串编码问题&rsquo;ascii&rsquo; codec can&rsquo;t encode&nbsp;characters</li> </ol> <h2 id="_2">调试发现:<a class="headerlink" href="#_2" title="Permanent link">&para;</a></h2> <ol> <li>print repr(中文参数1), 会打印\udc&nbsp;开头的而非\x开头的utf8型编码。</li> <li>比如&rdquo;《&rdquo; 正常编码是 <strong>&lsquo;\xe3\x80\x8a&rsquo;, 此处确是打印了&rsquo;\udce3\udc80\udc8a&rsquo;</strong>&nbsp;。</li> <li>改变逻辑,直接ssh到vps并执行 <strong>b中文.py 《xxx》</strong>&nbsp;没有问题!</li> </ol> <h2 id="_3">问题定位:<a class="headerlink" href="#_3" title="Permanent link">&para;</a></h2> <ol> <li>个人本机ubuntu系统测试不会出现bug,vps才出现,所以应该是shell环境或者是python环境问题。</li> <li>打印执行a.sh的shell环境,对比发现本机有<span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8和<span class="caps">LANGUAGE</span>=zh_CN:zh,vps仅有<span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8。</li> <li>把crontab中强加的环境变量<span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8去掉,此时a.sh的环境变量为<span class="caps">LANG</span>=en_US.<span class="caps">UTF</span>-8,&nbsp;vps恢复正常。(2小时排查出结果了!)</li> <li>总结: 之前觉得<span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8与<span class="caps">LANG</span>=en_US.<span class="caps">UTF</span>-8没什么不同,从此改观。</li> </ol> <h2 id="_4">问题解决:<a class="headerlink" href="#_4" title="Permanent link">&para;</a></h2> <p>去掉<span class="caps">LANG</span>=zh_CN.<span class="caps">UTF</span>-8,之后执行过程中会自动变成默认<span class="caps">LANG</span>=en_US.<span class="caps">UTF</span>-8</p> <h2 id="_5">原因探究:<a class="headerlink" href="#_5" title="Permanent link">&para;</a></h2> <p>待定</p> <h2 id="python-reprudc-print">python repr输出udc开头字符串, print(参数)导致异常<a class="headerlink" href="#python-reprudc-print" title="Permanent link">&para;</a></h2> <div class="highlight"><pre><span></span>&#39;/home/maskuser/path/to/ts/\udce3\udc80\udc8a\udce9\udc80\udc97.....mp4&#39; Traceback (most recent call last): File &quot;/home/maskuser/pathtodir/script/20181105\udce8\udca7\udc86\.....py&quot;, line 73, in &lt;module&gt; video_upload_testsite(*sys.argv[1:]) File &quot;/home/maskuser/pathtodir/script/20181105\udce8\udca7\udc86\.....py&quot;, line 29, in video_upload_testsite print (videopath) UnicodeEncodeError: &#39;ascii&#39; codec can&#39;t encode characters in position 27-50: ordinal not in range(128) </pre></div>