Programming and Scripting :: MyDSL info parser

Quote

I think your collapsable browser would be nicer than those two drop downs.

I thought about that yesterday morning, but at the time I didn't feel it was worth attempting until after making the updatedb function more reliable.

Quote

Xfree86-devel.unc is the only one that screws up.

Just noticed that it contains two "Current:" lines

Quote (mikshaw @ Mar. 16 2008,05:06)

Quote

I think your collapsable browser would be nicer than those two drop downs.

I thought about that yesterday morning, but at the time I didn't feel it was worth attempting until after making the updatedb function more reliable.

Quote

Xfree86-devel.unc is the only one that screws up.

Just noticed that it contains two "Current:" lines

Thanks for the update on XFree86-devel.unc. I have fixed it.

This is that sort of thing that I have been "cleaing up" to try to get data to be INSERTable to an SQL table. It has been more work that I originally thought.

Anyway, I have posted and updated mydslinfo.bz2

Quote

This is that sort of thing that I have been "cleaing up" to try to get data to be INSERTable to an SQL table.

I'm guessing that converting all dates to yyyy/mm/dd (if that's in your plan) is going to be a pain.

Here's a version that uses the collapsible browser rather than two separate menus.

There are a few things to consider....

It hasn't been thoroughly tested. There are probably issues I haven't seen yet.

In order for it to work as is, it needs the database to retain two existing qualities (besides the obvious keywords Location, Title, etc)
1) A line break between every package
2) The database has all apps together, all games together, etc. It currently compares the Location string to an existing variable to decide whether or not to create a submenu. If the categories get mixed in the database file, there will be multiple copies of some submenu headers.

It still needs some tweaks of the keyboard control. For example, after installation or database update, you need to either hit Tab a few times or switch to search and back to browse in order to get keyboard focus to the file list.

I don't yet know a way to change the sort order of the list, which is a feature I've had in the back of my mind since the beginning. I'm hoping to be able to allow the user to list files in alphabetical or chronological order (at least in the search window), and maybe have user-defined columns, like license, author, and maybe description.

I was also considering having only one category expanded at any one time, but for some reason that was giving me a lot more trouble than I had expected.

EDIT: fixed the browser doubling up after a db update.
Also looking into an easier way to keep track of which info table index is required at any given time rather than having to run down through it each time.

Code Sample

#!/home/dsl/bin/murgaLua-0.6.4
--[[
MyDSL info browser, 2008 mikshaw

Changelog (yyyy/mm/dd)
2008/03/16: Interface tweaks
2008/03/15: Improved the update function
Database date check uses os.date/lfs instead of os.popen/ls/awk
Use wizard instead of tabs for better use of resources
2008/03/14: Undid gmatch change due to improvements to the database file
Improved keyboard control
Use Fl_Menu_Button for menus to add to keyboard control
Fixed crash on search or install with an empty database
2008/01/24: Fixed string.gmatch to break up files even if there is no newline between them
2008/01/17: Added date of last update
Tweaked the update function (it's still sloppy)
Fixed ".mydsl_dir not found" error message
2008/01/16: Fixed resizable behavior
Uses file in .mydsl_dir
Added auto update for first run
2008/01/15: Added text search
2008/01/14: Fixed display of incorrect duplicate info file
Check for trailing spaces in title string
Added "Update" function
Code and interface tweaks
2008/01/13: start of project

--]]

mydsldirfile=io.open("/opt/.mydsl_dir")
if not mydsldirfile then
print(arg[0]..": Can't open /opt/.mydsl_dir")
os.exit(1)
end
mydsl_dir=mydsldirfile:read("*l")
mydsldirfile:close()

listfile=mydsl_dir.."/mydslinfo.bz2"
listurl="ftp://ibiblio.org/pub/Linux/distributions/damnsmall/mydsl/mydslinfo.bz2"
list_sep="@b@B50@u"

function toggle(b)
local count,state,one,two=1
-- selected line and its label
local me,label=b:value(),b:text(b:value())
if string.find(label,"%+%s*") then
state,one,two="expanded","%+ ","%- "
else
state,one,two="collapsed","%- ","%+ "
end
b:text(me,string.gsub(label,one,two)) -- swap + and -
-- toggle all lines until reaching the next section or the browser end
while b:text(me+count) and not string.match(b:text(me+count),list_sep) do
if state=="collapsed" then b:hide(me+count)
else b:show(me+count)
end
count=count+1
end
end

function fill_list()
inputfile = io.popen("bunzip2 -c "..listfile)
data=inputfile:read("*a")
inputfile:close()
if not string.find(data,"%w") then -- if there's nothing readable in it
info_display_buffer:text("Cannot read database "..listfile.."\nPlease try updating again.")
else
filelist:clear()
header={}
info={} -- table containing all package data: info[index].{location,title,date}
info_count=0 -- number of packages
local current_loc="" -- location string
local headers=0 -- nuber of locations
-- breaks up info for each package into a separate string
for s in string.gmatch(data,"(Location:.-)\nLocation:") do
info_count=info_count+1
info[info_count]={
location=string.match(s,"Location:%s*(.-)%s*\n"),
title=string.match(s,"Title:%s*(.-)%s*\n"),
date=string.match(s,"Current:%s*(.-%d)%s"),
text=s
}
if current_loc ~= info[info_count].location then
-- create a header when location changes
current_loc=info[info_count].location
filelist:add(list_sep.."- "..current_loc)
headers=headers+1
header[headers]=filelist:size()
end
filelist:add(info[info_count].title)
end
for i,v in ipairs(header) do filelist:value(header[i]); toggle(filelist) end
last_update:label("last update "..os.date("%d %b %Y",lfs.attributes(listfile).modification))
end
filelist:value(1)
tab_browse:setonly()
end

function updatedb()
-- download to temp file, check file integrity, and only then replace the old file
tempfile=os.tmpname()
os.execute("aterm +tr -bg white -fg black -geometry 80x4 -title \"Database Update\" -e wget "..listurl.." -O "..tempfile)
if os.execute("bzip2 -t "..tempfile) == 0 then
os.rename(tempfile,listfile)
fill_list()
else fltk.fl_alert("Database check failed.\nThis may be the result of an incomplete download\nor gremlins in your internets.")
os.remove(tempfile)
end
end

function pick_search_result()
if results:value() > 1 then
-- show selected info file in the browse tab
get_loc=string.match(results:text(results:value()),spacer.."(.*)"..spacer)
get_fname=string.match(results:text(results:value()),"(.-)"..spacer)
-- select the package in filelist
local myloc=1
while myloc<=filelist:size() do
if not string.find(filelist:text(myloc)," "..get_loc,1,1) then
myloc=myloc+1
else filelist:value(myloc)
break
end
end
-- open submenu if not open
if string.find(filelist:text(myloc),list_sep.."+ ",1,1) then toggle(filelist) end
while myloc<=filelist:size() do
if filelist:text(myloc) ~= get_fname then
myloc=myloc+1
else filelist:value(myloc)
break
end
end
for i=1,info_count do
-- make sure you get the right info AND category
if info[i].title == get_fname and info[i].location == get_loc then
info_display_buffer:text(info[i].text)
tab_browse:setonly()
myfile=info[i] -- this is the file that will be downloaded
break
end
end
tabs:value(tab1)
end
end

function list_cb()
if filelist:value()>0 then -- clicking a blank space can be fatal without this check
if Fl:event() == fltk.FL_RELEASE or Fl:event_key()==fltk.FL_Enter then -- on click or Enter
if string.find(filelist:text(filelist:value()),list_sep) then
toggle(filelist) -- if it's a separator
else
-- find out what submenu we're under
local myloc=filelist:value()
while myloc > 0 do
myloc=myloc-1
if string.find(filelist:text(myloc),list_sep) then
myloc=string.gsub(filelist:text(myloc),".-%s%s","")
break
end
end
for i=1,info_count do
-- make sure you get the right info AND category
if info[i].title == filelist:text(filelist:value()) and info[i].location == myloc then
info_display_buffer:text(info[i].text)
myfile=info[i]
tab_browse:setonly()
break
end
end
end
end
end
end

ww=500; wh=360; bh=30; bw=ww/2-5
win=fltk:Fl_Window(ww,wh,"MyDSL Info Browser")

tabs=fltk:Fl_Wizard(0,0,ww,wh-bh-5)
tab1=fltk:Fl_Group(0,0,ww,wh-bh-5,"browse")

tiles=fltk:Fl_Tile(5,5+bh,ww-10,wh-(15+bh*2))
filelist=fltk:Fl_Hold_Browser(5,5+bh,ww/2-5,wh-(15+bh*2))
filelist:callback(list_cb)
info_display=fltk:Fl_Text_Display(ww/2,5+bh,ww/2-5,wh-(15+bh*2))
info_display:textfont(fltk.FL_SCREEN)
info_display:textsize(12)
info_display_buffer=fltk:Fl_Text_Buffer()
info_display:buffer(info_display_buffer)
fltk:Fl_End() --end tiles

-- offscreen button that allows the list to respond to Enter key
enter=fltk:Fl_Return_Button(ww,wh,10,10)
enter:callback(function() filelist:do_callback() end)

install=fltk:Fl_Button(ww/2,5,bw,bh,"&Install Selected Package")
install:callback(
function()
if myfile then
mydslload=os.execute("mydsl-load "..myfile.title.." "..myfile.location)
if mydslload~=0 then fltk.fl_alert(myfile.title..":\nDownload or Checksum error!") end
end
end
)

b_updatedb=fltk:Fl_Button(5,5,bw,bh,"&Update Database")
b_updatedb:callback(updatedb)
fltk:Fl_End() --end tab1

tab2=fltk:Fl_Group(0,0,ww,wh-bh-5,"search text")
search_text=fltk:Fl_Input(85,5,ww-90,bh,"search text ")
search_text:when(fltk.FL_WHEN_ENTER_KEY)
search_text:callback(
function()
if search_text:value() ~= "" and info_count and info_count>0 then
results:clear()
if Fl_Browser.column_widths then
results:add("@B49@b@uPackage Name"..spacer.."@B49@b@uLocation"..spacer.."@B49@b@uRelease Date")
else
-- default screen font does not have bold or underline
-- and no columns means only one format string is used
results:add("@B49PACKAGE NAME"..spacer.."LOCATION"..spacer.."RELEASE DATE")
end
for i=1,info_count do
if string.find(string.lower(info[i].text),string.lower(search_text:value()),1,true) then
if not info[i].date then info[i].date="" end
results:add(info[i].title..spacer..info[i].location..spacer..info[i].date)
end
end
end
end
)

results=fltk:Fl_Hold_Browser(5,5+bh,ww-10,wh-(15+bh*2))
-- column widths available in 0.6.4+
if Fl_Browser.column_widths then
results:column_widths({results:w()/2,200,0})
spacer="\t"
else spacer="\t\t" -- force some extra space for older versions
results:textfont(fltk.FL_SCREEN) -- fixed-width makes it closer to columns
end
results:callback(
-- react to click only
function()
if results:value() > 1 and Fl:event()==fltk.FL_RELEASE then
pick_search_result() end
end
)
-- offscreen results_callback button for Enter key
results_cb=fltk:Fl_Return_Button(ww,wh,bh,bh)
results_cb:callback(pick_search_result)
fltk:Fl_End() --end tab2
fltk:Fl_End() --end tabs group

-- using buttons instead of tabs gives better keyboard control
tabbuttons=fltk:Fl_Pack(5,wh-bh,200,bh-5)
tabbuttons:type(fltk.FL_HORIZONTAL)
tabbuttons:spacing(5)
tab_browse=fltk:Fl_Radio_Button(0,0,100,bh-5,"&browse")
tab_browse:when(fltk.FL_WHEN_CHANGED)
tab_browse:box(fltk.FL_BORDER_BOX); tab_browse:selection_color(7)
tab_browse:callback(function() tabs:value(tab1) end)
tab_search=fltk:Fl_Radio_Button(0,0,100,bh-5,"text &search")
tab_search:when(fltk.FL_WHEN_CHANGED)
tab_search:box(fltk.FL_BORDER_BOX); tab_search:selection_color(7)
tab_search:callback(
function()
tabs:value(tab2)
Fl:focus(search_text)
end
)
fltk:Fl_End() --end tabs buttons
last_update=fltk:Fl_Box(bw+5,wh-bh-5,bw,bh+5)

win:resizable(tabs)
tabs:resizable(tab1)
tabs:resizable(tab2)
tab1:resizable(tiles)
tab2:resizable(results)
win:show()
find_file=io.open(listfile)
if find_file then data=find_file:read("*a") else data="" end
if string.find(data,"%w") then
find_file:close()
fill_list()
else updatedb()
end
Fl:run()

I handled all the different date formats with standard Lua captures while building and inserting to sqlite.

I think it only fair for me to show you mine since we may want to join some features especially if you are wanting to sort etc.

I started by using your GUI and stripping all flatfile code.
I added some "when" events and used "pack" on some layouts.
I stared by creating a simple database ignoring comments and other "multiline/multivalue" fields.
I then wrote standard lua and luasqlite as binding was available in murgaLua to query, display an index of 'hits' and then a view.
Later I moved the GUI around using a menu for update and quit.
Added the "comments" to database. Added QBE screen, still working on it.

This is a work in progress and I post it only as a preview. The next build of the database will break everything.

I post mainly for mikshaw but anyone can preview it. It currently only support direct sql queries. If you don't know sql don't preview it. The QBE is not implemented yet.

Of course you can download the mydsl.db open a shell start sqlite and issue sql commands too. murgaLua does not currently support readline in Linux so expect to do alot of CLI typing.

Open to suggestions on either the SQL database and the supporting GUI.

I am thinking that the multiline change-log and comments need to be separate tables.

I am not asking anyone to finish it as I am currently working on many parts. I start many sections of code before I "button" things up. That's just how I "work" er, play.

Oh, and this works with DSL 4.2.5 or DSL 3.4.11 no need to grab latest murgaLua

Code Sample

#!/bin/murgaLua
--[[
MyDSL Extension Browser, 2008 mikshaw - SQL and other GUI mods by Robert Shingledecker
2008/02/26: Added packed comments (probably needs sep table
2008/02/16: Added menu for Update database and Quit.
2008/02/14: Created QBE tab based on feedback not finished.
2008/02/08: Basic SQL queries working - take to Scale6x to show John.
2008/01/28: Dropped columns and switched to vertical VIEW layout.
2008/01/24: Created sqlite database mydsl.db with only simple cols.
2008/01/21: Gutted all flat file logic kept some GUI.
2008/01/13: Started from mikshaw's 2008 mydsl info browser.
]]--

function updatedb(nofile)
if not nofile then backup=os.rename(database,database..".bak") end
os.execute("aterm +tr -bg white -fg black -geometry 80x4 -title \"Database Update\" -e wget "..dburl.." -O "..database)
end

function doSQL()
if inputSQL:value() ~= "" then
results:clear()
selected = {}
field_names = {}
stmt = db:prepare(inputSQL:value())
if stmt == nil then
fltk.fl_alert(db:errmsg())
else
field_names = stmt:get_names()
for row in stmt:rows(inputSQL:value()) do
results:add(row[1].."\t"..row[2])
selected[#selected+1] = row
end
stmt:finalize()
end
end
end

function showSQL()
if results:value() > 0 then
viewResults = ""
for i = 1, #field_names do
if field_names[i] == "comments" then
selected[results:value()][i] = string.gsub(selected[results:value()][i],"%^","\n")
end
viewResults = viewResults .. field_names[i].."\t"
viewResults = viewResults .. selected[results:value()][i].."\n"
end
info_display:value(viewResults)
tabs:value(tab1)
tab1:hide()
tab3:show()
end
end

ww=420; wh=360; bh=25
w=fltk:Fl_Window(ww,wh,"MyDSL Extension Browser")
w:callback(
function(w)
db:close()
os.exit(0)
end)

mb = fltk:Fl_Menu_Bar(0,0,ww,bh)
mb:add("&File/&UpdateDB",0,updatedb)
mb:add("&File/E&xit",0,quit)
mb:callback(menu_items)

tabs=fltk:Fl_Tabs(0,bh,ww,wh-bh)

tab1=fltk:Fl_Pack(1,bh+1,ww,0,"SQL")
inputSQL=fltk:Fl_Input(0,0,ww-10,bh)
inputSQL:when(fltk.FL_WHEN_ENTER_KEY_ALWAYS)
inputSQL:callback(doSQL)

results=fltk:Fl_Select_Browser(0,0,ww-10,wh-(8+bh*3))
results:textfont(fltk.FL_SCREEN)
results:textsize(12)
results:when(fltk.FL_WHEN_CHANGED)
results:callback(showSQL)
fltk:Fl_End()

tab2=fltk:Fl_Group(0,0,ww,wh-bh,"QBE")
field_label = {"loc","title","desc","ver","author","site","policy","user","current","comments"}
field_len = {120,180,325,240,320,290,325,285,110,335}
field_in = {}
for i = 1, #field_label do
field_in[i] = fltk:Fl_Input(75,28+(28*(i-1)),field_len[i],bh,field_label[i])
end
okBtn = fltk:Fl_Return_Button(340, 310, 70, 20, "OK");
okBtn:callback(
function(okBtn)
for i = 1, #field_label do
if field_in[i]:value() ~= "" then
print(field_label[i] .." ".. field_in[i]:value())
end
end
end)
fltk:Fl_End()

tab3=fltk:Fl_Group(0,0,ww,wh-bh,"View")
info_display=fltk:Fl_Multiline_Output(4,bh+4,ww-10,wh-(30+bh*2))
info_display:textfont(fltk.FL_SCREEN)
info_display:textsize(12)

install=fltk:Fl_Return_Button(340,310,70,20,"Install")
install:callback(
function()
os.execute("mydsl-load "..selected[results:value()][2].." "..selected[results:value()][1])
end
)

w:resizable(tabs)
tabs:resizable(tab1)
tabs:resizable(tab2)
tab1:resizable(info_display)
tab2:resizable(results)
w:show()

mydsldirfile=io.open("/opt/.mydsl_dir")
if not mydsldirfile then
print("can't open "..mydsldirfile)
os.exit(1)
end
mydsl_dir=mydsldirfile:read("*l")
mydsldirfile:close()

database = mydsl_dir.."/mydsl.db"
dburl="ftp://ibiblio.org/pub/Linux/distributions/damnsmall/mydsl/mydsl.db"
databaseCheck = io.open(database)
if databaseCheck then
databaseCheck:close()
else
updatedb(1)
end

selected = {}
field_names = {}
db = sqlite3.open(database)
inputSQL:value("select * from mydsl order by loc;")

Fl:run()

Thanks for sharing. I'll try it very soon.

Quote

Oh, and this works with DSL 4.2.5 or DSL 3.4.11 no need to grab latest murgaLua

Are you implying there is trouble with my experiment beyond the lack of columns in murgaLua versions prior to 0.6.4? I tried to make sure it works in 0.5.5, but maybe I missed?

Next Page...
original here.